All posts tagged polls

Book review: The Signal and the Noise: Why So Many Predictions Fail-but Some Don’t by Nate Silver.

This is a well-written book about the challenges associated with making predictions. But nearly all the ideas in it were ones I was already familiar with.

I agree with nearly everything the book says. But I’ll mention two small disagreements.

He claims that 0 and 100 percent are probabilities. Many Bayesians dispute that. He has a logically consistent interpretation and doesn’t claim it’s ever sane to believe something with probability 0 or 100 percent, so I’m not sure the difference matters, but rejecting the idea that those can represent probabilities seems at least like a simpler way of avoiding mistakes.

When pointing out the weak correlation between calorie consumption and obesity, he says he doesn’t know of an “obesity skeptics” community that would be comparable to the global warming skeptics. In fact there are people (e.g. Dave Asprey) who deny that excess calories cause obesity (with better tests than the global warming skeptics).

It would make sense to read this book instead of alternatives such as Moneyball and Tetlock’s Expert Political Judgment, but if you’ve been reading books in this area already this one won’t seem important.

A number of people have compared the final forecasts for the election (e.g. this), but I’m more interested in longer term forecasting, so I’m comparing the state-by-state predictions of Intrade and FiveThirtyEight on the dates for which I saved FiveThirtyEight data a month or more before the election.

Here is a table of states where Intrade disagreed with FiveThirtyEight on one of the first four dates for which I saved FiveThirtyEight data or where they were both wrong on July 24. The numbers are probability of a Democrat winning the state’s electoral votes, with the Intrade forecast first and the FiveThirtyEight forecast second.

State 2008-07-24 2008-08-22 2008-09-14 2008-10-01
CO 71/68 60/53 54.5/46 67.5/84
FL 42/29 34.5/28 30/14 55.2/70
IN 38/26 34.1/15 20/11 38/51
MO 50/26 32.9/13 22.1/11 42.5/48
NC 30/22 25/21 14/7 51/50
NV 51.2/49 49/45 44.9/32 55/66
OH 65/53 50/38 40/29 53.5/68
VA 60.5/50 52.3/36 42/22 59/79

On July 24, both sites predicted Florida, Indiana, and North Carolina wrong. FiveThirtyEight got Indiana right on Oct 1 when Intrade was still wrong, but Intrade got North Carolina right on that date (just barely) while FiveThirtyEight rated it a toss-up.
Intrade got Nevada right on July 24 (just barely) while FiveThirtyEight got it wrong (just barely).
For Virginia, Intrade was right in July and August while FiveThirtyEight was undecided and then wrong.
FiveThirtyEight got Colorado wrong on September 14, but Intrade didn’t.
FiveThirtyEight got Ohio wrong on August 22, while Intrade got it right.
Intrade rated Missouri a toss-up on July 24, while FiveThirtyEight got it right.

On September 14, FiveThirtyEight was fooled by McCain’s post convention bounce by a larger margin than Intrade, but by Oct 1 FiveThirtyEight was more confident about correcting those errors.
For states that were not closely contested, there were numerous examples where Intrade prices where closer to 50 than FiveThirtyEight. It’s likely that this represents long-shot bias on Intrade.

In sum, Intrade made slightly better forecasts for the closely contested states through at least mid September, but after that FiveThirtyEight was at least as good and more decisive. Except for Intrade’s Missouri forecast on July 24, the errors seem largely due to underestimating the effects of economic problems – errors which were also widespread in most forecasts for other things affected by the recession.

In the senate races, I didn’t save FiveThirtyEight forecasts from before November 1. It looks like both Intrade and FiveThirtyEight made similar errors on the Alaska and Minnesota races.
[Update on 2009-01-13: contrary to initial reports, they apparently got the Alaska and Minnesota races right, although there’s still some doubt about Minnesota.]

On the other hand, Intrade had been fairly consistently (but not confidently) saying since early July that California’s Proposition 8 (banning same-sex marriage) would be defeated. Pollsters as a group did a somewhat better job there by issuing conflicting reports.

To deter any suspicion that the comparisons I plan to make between Intrade’s predictions and polls are comparisons I selected to make Intrade look good, I’m announcing now that I intend to use FiveThirtyEight.com as the primary poll aggregator. I intend to pay attention to predictions that are more long-term than I focused in 2004, so the comparison I’ll attach the most importance to will be based on the first snapshot I took of FiveThirtyEight.com’s state by state projections, which was on July 24.

Also, as of last week, one of the Presidential Decision Markets that I’m subsidizing, DEM.PRES-OIL.FUTURES, has attracted enough trading (I suspect from one large trader) to make me reasonably confident that it’s showing the effects of trader opinion rather than the effects of my automated market maker (saying that oil futures will drop if the Democratic candidate wins, and rise if he loses).

Once again, I feel somewhat humbled for underestimating the accuracy of presidential election markets. At least I was cautious enough to mainly bet against Bush winning states where he appeared to be behind, and against him winning 400 electoral votes, which made up for what I lost betting that Kerry would win the election and popular vote.

Assuming the preliminary results are accurately indicating the final results, Tradesports did quite well at predicting the elections (except for a few hours on Tuesday afternoon when it mistakenly reacted to exit polls). It’s Monday evening prices correctly indicated which presidential candidate would win each state. And it did a good job of indicating which states were closest (saying Iowa and Ohio were the least certain).

Continue Reading