Condorcet, Hugo, and sad puppies

Yesterday, this year's Hugo Award winners were announced; this is an annual fan popularity contest for the best works in science fiction and fantasy (there is a different set of awards voted on by the writers themselves, the Nebulas). I have a few thoughts on the nominees (like, why wasn't Her among them?) but that's not what I'm writing about. Rather, what interests me in this year's contest is the issue of voting systems and their resistance to manipulation.

Some background: this year's award nomination and voting involved a group of fans from one of the subgenres of SF whose two main interests seem to be military jingoism and sexual and racial anti-inclusivity and who have apparently dubbed themselves the "sad puppies". These people pushed a slate of nominees onto the ballot, which then lost fairly decisively in the final voting. There was no unfair vote manipulation going on: everything was aboveboard and according to the rules. But it caused me to wonder: how large would a dedicated faction of the voters have to be to break into the winner's circle, against the will of the remaining voters?

To model this we need to know something about how the voting worked and we need to make some assumptions (not visible in the actual Hugo voting data) about the details of the voter preferences. In the nomination stage, any eligible voter may cast one vote in any category. The top five vote winners (after removing ineligible candidates, candidates that received less than 5% of the total nominations, and candidates who decline the nomination) become the short list for the final voting. If there is a tie for fifth place, the tied candidates are all included. In the final voting, each eligible voter makes an ordered list of their favorite candidates, in order from first, second, etc. They are not required to list all candidates in this list, are allowed to list a special placeholder candidate called "no award", and are also allowed to list other candidates after no award. These preferences are aggregated using an algorithm (described below) to produce a winner for each category of the awards. I'm going to assume (without justification) that the sad puppies prefer their candidates, then no award, then all other candidates, while the other voters have the reverse preferences: all other candidates (in some combination of orders that I am not going to make assumptions about), then no award, then the sad puppy candidates.

So, first, how easy is it to get your candidate on the short list of nominees? Pretty easy, it turns out. If the sad puppy faction were only 20% of the electorate, they would be able to guarantee themselves a place on the short list even if the remaining voters somehow conspired to make it as hard as possible for them. The actual cutoff was much lower, ranging from 5.0% in the best short story category to 13.9% in the best long-form dramatic presentation category. The percentages achieved by the sad puppies ranged from 9.5% to a little higher, enough to secure their nominations.

Next, and more complicatedly, how easy is it for the sad puppies to actually win? The answer turns out to be: much harder. One indication of this is given by something called the Condorcet criterion for preference balloting. This states that, if one candidate would win all head-to-head contests then that candidate should win the overall election; if this is the case, then an election system is immune to certain kinds of manipulation by minority factions. Unfortunately there are two problems with this analysis for the Hugos: first, the assumptions I've made don't imply the existence of a Condorcet winner. There could be a directed cycle of head-to-head winners among the non-sad-puppy candidates, even though each of them individually would win a head-to-head contest against a sad puppy. When this happens, the Condorcet criterion says nothing about who the winner should be. A generalized Condorcet property fixes this: if the candidates can be partitioned into two sets A and B such that every candidate in A wins a head-to-head contest against every candidate in B, then the winner should be from A. A system that satisfies the generalized Condorcet property would never pick a sad puppy unless the sad puppies had an outright majority of the voters.

But the second problem with trying to apply the Condorcet analysis to this situation is that the election system used for the Hugos does not obey the Condorcet criterion (in either the single-winner or generalized forms). There are systems that do have the Condorcet property; my favorite is the Schulze method, which involves computing widest paths in a complete directed graph describing the head-to-head results between all pairs of candidates. If the Hugo awards used this method, the Condorcet criterion would ensure their immunity to sad puppies. But the method they actually use is instant-runoff voting. This method proceeds in a sequence of rounds, eliminating one candidate per round until a winner is left. In each round, each voter's vote is given to the remaining candidate that is highest on the voter's list (or the vote is discarded if no listed candidate remains) and the candidate with the fewest votes is kicked off the island. But this is not a Condorcet system; for instance, a candidate that is every voter's second choice would be eliminated in the first round (because that candidate gets no first place votes) but could easily be a Condorcet winner (if no other candidate gets an outright majority of first place votes).

Instant-runoff does satisfy a different property, the majority criterion: anyone who gets an outright majority of first-place votes will necessarily win, because their number of votes can only improve after other candidates are eliminated. This still doesn't help against the sad puppies, because outright majorities are unlikely (in the 2014 Hugos, it happened only for the fan artist category). But we can generalize it just like we can generalize the Condorcet criterion. Call the "generalized majority criterion" the following property: if the candidates can be partitioned into two sets A and B such that a majority of the voters thinks all candidates in A are better than all candidates in B, then the winner should be a candidate in A. Whenever a voting system satisfies the generalized Condorcet criterion it also satisfies the generalized majority criterion. Instant-runoff also satisfies the generalized majority criterion, because once we reach a round in which all but one of the candidates in A has been eliminated, the remaining candidate in A will have an outright majority and will win all remaining rounds. Using any voting system that obeys the generalized majority criterion, and with the assumptions about sad puppy voting patterns made above, the sad puppies can't win without an outright majority of the voters. If the sad puppies are not a majority, then a majority of voters agrees with the partition in which B is the sad puppy candidate and A is everybody else. In particular, the sad puppies can't win an instant runoff without a majority. The existence of "no award" doesn't really make much difference to this analysis: it would be valid with or without the ability to list no award.

My colleague Don Saari is an expert on voting systems and a strong advocate of a different preference balloting system, the Borda count. This system is easier to explain than instant-runoff (and much easier than Schulze): if there are six candidates (counting no award) we give five points for each first place vote, four for each second place vote, etc., down to one point for a second-to-last vote, and give the award to whoever gets the most points. How does this method fare against the sad puppies? Not so well. If the other candidates are close to equal in strength, then their voters will split their votes evenly among 5, 4, 3, and 2 points, with 1 for no award. The average number of points per candidate will be 3.5x where x is the number of non-sad-puppy voters. On the other hand the sad puppies will vote their candidate first, and then (if they're trying to get the strongest result for this candidate) omit listing anybody else to deny them the points they would get for lower finishes. This would give the sad puppies 5y points where y is the number of sad puppy voters. If 5y > 3.5x (that is, if the sad puppies have a bit more than 41% of the electorate) then they have a chance of winning. In the limit of large numbers of candidates, even a 33% minority of sad puppies could be enough to swing an election. That is, although the Borda count has some robustness against small minority factions, it is more vulnerable to large minority factions than Schulze or instant runoff.

As for which election system to choose: it depends on what you want winning to mean. The system you should use for a popularity contest such as this one could well be different than the system you would prefer for a political office. Is it better for the winner to be somewhat liked by most voters or to have strongly enthusiastic support by a smaller number of voters? The instant runoff system used by the Hugos demands a balance of both: a winner needs enough enthusiasm to make it through the early rounds of voting and enough depth of support to make it through the late rounds.

---

Comments: