Critics Love 'Social Network.' Will The Academy Defriend It ...

Nate Silver will be providing awards-season insight for Carpetbagger leading up to the 83rd Academy Awards on Feb. 27. Mr. Silver is the author of the Times’ FiveThirtyEight blog, which is devoted to the analysis of statistics and data in politics and other areas.

Last year, the Academy of Motion Picture Arts and Sciences adopted instant-runoff voting for its selection of best picture, a decision which is sometimes credited for its choice of “The Hurt Locker” over “Avatar” and other nominees. Although the system has its critics, it’s no more convoluted than, say, the voting process for “Dancing With The Stars.” Let me walk you through it:

Instead of simply voting for one candidate, each voter instead ranks all of the choices on his ballot from top to bottom (meaning, in this case, from 1st to 10th). The first-place votes are then tallied. If no choice has received an outright majority of first-place votes, then the choice with the fewest first-place votes is eliminated. If your first choice is eliminated, then your second-place choice gets your vote instead. For instance, if you had ranked “Little Fockers” first, and nobody else had agreed with you that it deserved best picture, your second choice (say, “Twilight: Eclipse”) takes its place. This process then repeats itself, with the candidate with the fewest first-place votes being dropped one stage at a time until one of the remaining choices has received an outright majority.

Still confused? Perhaps it’s the sort of thing best explained by example. Let’s use a data set, in fact, that might give us a few hints about how instant-runoff voting might play out in this year’s balloting: a database of critics’ reviews of the most likely nominees.

(No, critics’ picks do not always mirror those of the Academy, although the correspondence has been stronger of late: the Broadcast Film Critics Association Award for best film has matched the Academy’s pick in 9 of the past 11 years. But there’s a lot of data as to what critics thought about this years’ potential nominees, whereas there is very little of it as to the preferences of Academy voters. Exactly how informative critics’ picks are in predicting the Academy Award winners is a subject that we’ll be exploring in more detail in the coming days. In the meantime, this column may be best viewed as a “thought experiment” that sheds light on the instant-runoff voting process.)

Metacritic is one of several Web sites that aggregates critics’ reviews of films and other media. Unlike its primary competitor, Rotten Tomatoes, which simply categorizes reviews as good (“fresh”) and bad (“rotten”), Metacritic assigns each review a score from 0 to 100. This can sometimes make a fairly big difference. “The Social Network” and “True Grit” received similar ratings at Rotten Tomatoes, for instance, with “The Social Network” getting 97 percent “fresh” reviews and “True Grit” 95 percent.

While nearly everyone agreed that “True Grit” was a pretty good film, a lot more critics thought that “The Social Network” was not just good, but great. Metacritic determined that 28 critics had given it a perfect score of 100, whereas that was true of just one critic (The Times’s own Manohla Dargis) who had reviewed “True Grit.” Thus, “The Social Network” received a Metacritic rating of 95, significantly better than the 80 points that “True Grit” scored.

We can also use the Metacritic scores to create a best picture ballot for each of the critics, ranking their choices from first to tenth. We’ll then simulate an instant-runoff process to see who the critics would pick as best picture under the Academy’s rules.

Problem: the Academy Award nominees haven’t been announced yet â€" we’ll have to wait until Tuesday morning for that. But let’s make some educated guesses about which films might get picked.

It’s almost certain that the five films nominated for best dramatic picture by the Golden Globes will be among the Academy’s choices: “The Social Network,” “Black Swan,” “The Fighter,” “Inception” and “The King’s Speech.” “The Kids Are All Right,” the Golden Globe winner for best motion picture in its comedy and musical category, will also almost certainly be among the selections. So, very probably, will be “True Grit,” which the Golden Globes snubbed. And “Toy Story 3? is another likely nominee, as a tribute to the successful franchise that had the highest-grossing picture of 2010.

The last two choices are a little trickier to figure. But let’s go with two gutsy indie films â€" “Blue Valentine” and “Winter’s Bone,” each of which have received plenty of critical praise â€" to round out the Academy’s portfolio.

We now have our 10 films, but we have another, more technical problem: not all of the critics reviewed all of the movies. (You think the Academy’s voters have seen all of them either?) We’ll work around this by restricting the analysis to those critics who reviewed at least half of them â€" this leaves us with a 40-person panel â€" assigning a lukewarm score of 65 to any movies that the critic bypassed.

Yet another problem: it was quite common for one or more of the critic’s choices to have gotten the same Metascore. (A meta-criticism of Metacritic: they are a bit too generous about handing out scores of 100; there is a difference between a movie that a critic cannot find fault with and one that she thinks is a masterpiece.) We’ll break these ties by drawing lots.

Here, for example, is the ballot we’ve created for The Chicago Reader’s J.R. Jones with all of his ties broken (Metacritic scores are in parentheses).

1. “The Social Network” (100)
2. “The Kids Are All Right” (100)
3. “Winter’s Bone” (80)
4. “True Grit” (80)
5. “Black Swan” (70)
6. “The Fighter” (70)
7. “The King’s Speech” (70)
8. “Blue Valentine” ((not reviewed; default rating of 65 applied)
9. “Toy Story” ((not reviewed; default rating of 65 applied)
10. “Inception” (50)

Having established our ground rules, we’re now ready to count the votes from from Mr. Jones and the other 39 reviewers â€" and here is our tally:

Instant-runoff voting simulation â€" Stage 1
“The Social Network” (11 votes)
“Winter’s Bone” (6 votes)
“Toy Story 3? (5 votes)
“The Kids Are All Right” (4 votes)
“Black Swan” (4 votes)
“The King’s Speech” (4 voters)
“Blue Valentine” (3 votes)
“Inception” (2 votes)
“The Fighter” (1 vote)
“True Grit” (0 votes)

“The Social Network” is in first place, although it’s well short of the 21 votes that it would need to win a majority. So the tallying process continues â€" we must eliminate one of the films. That would be “True Grit”, which got no first place votes in our simulation.

A quick word about movies like “True Grit” and those like it before we boot it off the island. It is sometimes thought that an instant-runoff process benefits those films that are uniformly well-liked at the expense of those that some people love and some loathe. This is why, for instance, it can be argued that that “The Hurt Locker” â€" which received more consistent but less enthusiastic acclaim â€" was selected for Best Picture ahead of “Avatar” last year, which seemed to evoke more polarized reactions.

That might be true up to a point. But if a film were literally everyone’s second choice, it would be eliminated in the first round of vote-counting since the eliminations are based not which candidate received the most last-place votes, but rather (as on “American Idol”) which one got the fewest first-place ones.

This is very much the situation that “True Grit” finds itself in: it was rated considerably better on average than movies like “Inception” and “Black Swan.” But (almost) nobody thought it was the best film of the year. In general, receiving consistent (but not spectacular) reviews is an asset to a film late in the vote-tallying process, when it will begin to pick up votes as a compromise choice as others are eliminated. But it can be vulnerable early on in the counting if it doesn’t rank at the top of at least some ballots.

This is definitely a flaw with instant-runoff voting, by the way: hypothetically, a film that received the highest ranking on average  ballot (say that literally everyone voted it in second place, and the first place votes were split between many different candidates) could be the first one eliminated. But political scientists have determined there is literally no such thing as a perfect voting system â€" it’s just a matter of picking which flaws you’re willing to tolerate.

So now on to our second stage of vote-counting, which we find looks very much like the first:

Instant-runoff voting simulation â€" Stage 2
“The Social Network” (11 votes)
“Winter’s Bone” (6 votes)
“Toy Story 3? (5 votes)
“The Kids Are All Right” (4 votes)
“Black Swan” (4 votes)
“The King’s Speech” (4 voters)
“Blue Valentine” (3 votes)
“Inception” (2 votes)
“The Fighter” (1 vote)
“True Grit”

Since “True Grit” received no first-place votes, there are not votes to redistribute.

Instead, we’ll need to eliminate the next-lowest choice, which was “The Fighter.” It’s one first-place vote â€" from The Orlando Sentinel’s Roger Moore â€" goes to “Winter’s Bone” instead:

Instant-runoff voting simulation â€" Stage 3
“The Social Network” (11 votes)
“Winter’s Bone” (7 votes)
“Toy Story 3? (5 votes)
“The Kids Are All Right” (4 votes)
“Black Swan” (4 votes)
“The King’s Speech” (4 voters)
“Blue Valentine” (3 votes)
“Inception” (2 votes)
“The Fighter”
“True Grit”

“Inception” is the next film to go: both of its votes go to another youthful movie, “The Social Network.”

Instant-runoff voting simulation â€" Stage 4
“The Social Network” (13 votes)
“Winter’s Bone” (7 votes)
“Toy Story 3? (5 votes)
“The Kids Are All Right” (4 votes)
“Black Swan” (4 votes)
“The King’s Speech” (4 voters)
“Blue Valentine” (3 votes)
“Inception”
“The Fighter”
“True Grit”

You should be getting the hang of this by now. “Blue Valentine” departs us next. Two of its votes are reallocated to “The King’s Speech” â€" good for “The King’s Speech,” because it had been looking quite vulnerable with just four first-place votes.

Instant-runoff voting simulation â€" Stage 5
“The Social Network” (13 votes)
“Winter’s Bone” (7 votes)
“The King’s Speech” (6 voters)
“Toy Story 3? (6 votes)
“The Kids Are All Right” (4 votes)
“Black Swan” (4 votes)
“Blue Valentine”
“Inception”
“The Fighter”
“True Grit”

We now have a two-way tie for last place between “Black Swan” and “The Kids Are All Right.” It’s not clear how the Academy would handle such a situation, but we’ll eliminate “Black Swan” since it received a lower Metacritic score on average. Three of its four votes go to “The King’s Speech,” which is beginning to pick up steam:

Instant-runoff voting simulation â€" Stage 6
“The Social Network” (13 votes)
“The King’s Speech” (9 voters)
“Winter’s Bone” (8 votes)
“Toy Story 3? (6 votes)
“The Kids Are All Right” (4 votes)
“Black Swan”
“Blue Valentine”
“Inception”
“The Fighter”
“True Grit”

“The Kids Are All Right,” which had received a pardon last time around, now gets the death penalty. Oddly â€" I wouldn’t think that the audiences for the films would overlap â€" three of its four votes go to “Toy Story 3.”

Instant-runoff voting simulation â€" Stage 7
“The Social Network” (14 votes)
“The King’s Speech” (9 voters)
“Toy Story 3? (9 votes)
“Winter’s Bone” (8 votes)
“The Kids Are All Right”
“Black Swan”
“Blue Valentine”
“Inception”
“The Fighter”
“True Grit”

The next picture eliminated is “Winter’s Bone,” which leaves us with just three remaining choices:

Instant-runoff voting simulation â€" Stage 8
“The Social Network” (17 votes)
“The King’s Speech” (12 voters)
“Toy Story 3? (11 votes)
“Winter’s Bone”
“The Kids Are All Right”
“Black Swan”
“Blue Valentine”
“Inception”
“The Fighter”
“True Grit”

“The Social Network” now has 17 votes; it needs just four more to secure a majority. “The King’s Speech” has hung on to second place and survives the round, but it would need nine of the remaining 11 votes from “Toy Story 3? â€" which has had a good run but now has to go.

I should probably have rigged the example to produce a big comeback win for “The King’s Speech” (if the Academy were to reveal the voting results one stage at a time like this, there are possibilities under which it could be quite exciting). But instead, it’s “The Social Network” that clinches victory with a total of 23 votes.

Instant-runoff voting simulation â€" Stage 9
“The Social Network” (23 votes) â€" Best Picture winner
“The King’s Speech” (17 voters)
“Toy Story 3?
“Winter’s Bone”
“The Kids Are All Right”
“Black Swan”
“Blue Valentine”
“Inception”
“The Fighter”
“True Grit”

Our simulation finds that, if movie critics had been using the same system to vote for Best Picture as the Academy does, “The Social Network” would likely win Best Picture. That is no surprise, really: it has already won the Critics’ Choice Award. (It also received 2010’s highest Metacritic score.)

As I mentioned at the outset, exactly how much predictive power critics’ picks have in telling us how the Academy will vote remains an open question. What this exercise ought to suggest, however, is that if there is a sufficient amount of support for “The Social Network”, there is nothing about the instant-runoff voting that would prevent it from winning an award it would otherwise deserve.

Instant-runoff voting can make a difference when there is a choice between an “agreeable” candidate and one that some people love but other people can’t stand, perhaps tipping the balance toward the former choice. This may have been the situation last year, when we had a somewhat weaker field overall.

“The Social Network” is much more than “agreeable”, though: yes, nearly everyone likes the movie, but also, some people think it’s absolutely epic. In our experiment, it got to both have its cake and eat it too, picking up a lot of first-place votes at the outset, but also serving as a failsafe for many voters once other films fell by the wayside. If its critical reviews are any guide, it needs to be considered the favorite to win Best Picture, and perhaps a prohibitive one.