Pollster Guru Nate Silver's Forecasts of GOP Gains Has Bizzare Margin of Error

[bigtree]

from Nate Silver on the NYT FiveThirtyEight blog: http://fivethirtyeight.blogs.nytimes.com/2010/10/10/number-of-competitive-house-races-doubles-from-recent-years/?hp


The House forecast that we released on Friday establishes an over-under line for Republican gains at a net of 47 or 48 seats. But, as I noted at the end of the article, the confidence interval on this forecast is very wide. Its margin of error is about ±30 seats — meaning that a gain of as few as 17 seats, or as many as 78, is entirely possible — and there is a small chance of even larger or smaller gains.

When I noted this on Twitter on Friday, I got a few sarcastic replies: what good is a forecast if it tells you that essentially anything can happen . . .?

____ According to just about every objective and subjective indicator, then, the number of competitive House districts is roughly twice as high as in recent years. This is why the margin of error on our House forecast is very wide. If the polling is off by just a little in one direction or another, it could have profound consequences for the number of seats that Republicans are likely to gain. Likewise, there are a great number of districts in which both parties have viable candidates who could overperform or underperform the trends present in the national environment . . .


read more: http://fivethirtyeight.blogs.nytimes.com/2010/10/10/number-of-competitive-house-races-doubles-from-recent-years/?hp

[hlthe2b]

1. As those of us who use statistics know...

"garbage in, garbage out" ....

His forecasting models are dependent on other's polls and some of those polls are extremely week methodologically. If he throws out the worst, he has less data to work with and thus more "uncertainty."

But, i take it as a good thing, right now. "ANYTHING CAN HAPPEN".. indeed..;)

[hack89]

4. According to the link, the reason is simple

more close races than in the past. With 85 races within 10 points, to say there is a large amount of uncertainty in the results is an understatement.

[hlthe2b]

8. The closer the race, the more numbers you need to include in polls

to gain any kind of certainty. But, with the emphasis on fast results, rather than precision, pollsters are not going to double, triple (or more) the size of their sample.

[hack89]

11. How do you know that for a fact?

is that simply your opinion?

[hlthe2b]

20. No, it is a basic statistical principal..

Stated fairly simply, sample size required increases as the ability to discriminate between groups diminishes. In this case the two groups are nearly equally split in terms of their preferences, as opposed to, say, 20% of the group going one way and 80% going the other. In the case where the groups are closely split, a much larger sample size can decrease the probability of a "chance" finding--i.e., the uncertainty. A basic stats book chapter on sample size, power and statistical confidence will present this in greater detail.

[hack89]

23. You misunderstood my question

can you show me the sample sizes for the polls? What if a greater sample size showed in finer detail an uncommonly large number of very close races - the margin of error would still be large.

[hlthe2b]

24. Yeah, right. they share all their raw data with me..

I have worked with pollsters and I know how they are done when conducted by the best and conversely by those in it for the buck. Increasing sample size increases the cost of conducting the poll exponentially.

Believe what you wish. I have no desire to argue with you since you obviously know more than I. :sarcasm:

[hack89]

25. I don't know more than you

I just don't like unsubstantiated appeals to authority. If you can't back up your assertion why make it?

[hlthe2b]

26. I present it as one who has worked in the field

I do not claim to have behind the scenes access to these polls raw data any more than Nate does. It is very closely protected and proprietary data. And your rudeness is over the top. Unbelievable. I was just trying to offer some insight, as you would do in a field in which you are more versed than the average DUer. Incredibly rude, hack89.

[hlthe2b]

31. I'm talking about sample size within each poll...

You could add more and more and more polls, all conducted with small sample sizes and thus all with individual high margins of error and Nate's model will still be shit. You add a couple of exceedingly well done polls with very large samples sizes which allows for a reduction in the margin of error for each (and conversely throw out the worst conducted of the smaller polls) and Nate's margin of error would likewise be down and his mode would be far more precise. The problem is, is that he doesn't have that luxury. Thus, garbage in, garbage out.

Nate's methods are not the issue. It is that he is using other's polls to analyze and thus dependent on how well they have conducted and their own margins of error.

[hack89]

2. How can it be bizarre

when he goes into great detail why it is so high? His point is that there are many more close races than in the past - 85 races within 10 points.

[bigtree]

3. it's bizzare

because the margin of error is so high.

[hack89]

5. But he gives a detailed and rational explanation

it may be unique but I don't think bizarre is the proper description. Do you see any flaw in the rational or calculations?

[bigtree]

9. it's probably my own bias

. . . but I don't see why anyone should get away with using a poll (forecast) which has a 30 point margin of error to predict a republican takeover of the House (as many have).

[hack89]

13. Did you even read your own link?

he didn't use a single poll. What he said is that hundreds of individual polls show an uncommonly high number of close races because there are few incumbents without challengers. Logically it makes sense that you would be left with a large margin of error.

[bigtree]

16. jesus

so I called it a 'poll'. It's his forecast, or whatever you want to call it, and it's being used by him and others to predict a republican takeover. It's a dubious strategy for predicting the balance of power and I think it's more than a little misleading.

[hack89]

17. No - he is describing a fluid situation that Democrats can benefit from

if they are smart. He is describing an election balanced on a knife edge - if he was certain there would be a much lower margin of error. What is wrong with a pollster saying there is too much uncertainty to make an accurate forecast? He should be punished for being honest?

[bigtree]

19. I have no doubt that Democrats can benefit from the current political environment

I think Silver's work in collecting all of these polls is outstanding - I just don't hang on his predictions (specifically, his prediction of a republican gain of about 47 or 48 seats which he dutifully offers up every time he's interviewed).

[Me.]

6. My Prediction

15 (give or take) loss in house, dems also keep senate.

[Gman]

7. It appears to use the same methodology that TIA uses and it's meaningless

it's using known data with all it's flaws to forecast something for election day that the data implies right now, this moment. Polling is a snapshot in time. Not a future viewing movie. Nothing else, including what will happen on election day. Polling data evolves over time and the method used doesn't have the luxury of what the data will be in the future in order to be more accurate. Therefore, the simulation is meaningless.

BTW, TIA is TruthIsAll, a now tombstoned DU'er.

[DemocratSinceBirth]

30. It's Different

Nate includes all the polls on the assumption that the good ones will cancel out the bad ones. Amazingly TIA only included pollsters he likes.

Who knows if he will be right. I do know Larry Sabato who nailed 06 and 08 and teaches in one of the best and most respected political science programs in the nation is predicting pretty much the same thing. Intrade which also is right most of the time is aligned with Sabato and Silver.

[Gman]

34. Sabato and others are good, no doubt about it

but they make their predictions in different ways than these simulations.

This type of simulation is good for determining a product defect rate, or assessing risk in a system using known variables (e.g. calculating the rate of failure of surgical staples and the resulting monetary damages payout and if it meets an acceptable standard). But shooting random variables into equations with constraints that are fixed right now doesn't imply anything about what those constraints will look like on election day.

I just can't believe the NY Times ran with this.

[Mefistofeles]

10. Silver is covering his ass

If Dems lose only 17, he was right. If they lose 78, he was right too! And anything in between.

[Spazito]

12. If the margin of error is so big, why publish the results at all?

With such a big margin of error, the poll says nothing of substance, imo, and Silver would have better off stating because of the 'unknowns' polling results at this point are all over the map.

[hack89]

15. It says a lot - mainly that there are many close races

and therefore there is hope for either side. It show a fluid situation that presents an opportunity for Democrats.

I see this as good news.

[Spazito]

21. I agree, it is good news, my beef is that he first posted his forecast without...

stating the huge margin of error to go along with it. The header was: Projected Republican Gains Approach 50 House Seats

There was NO qualifier in that article, instead this is what he published:

"Our model now estimates that the Republicans have a 72 percent chance of taking over the House, up from 67 percent last week. Moreover, they have nearly even odds of a achieving a net gain of 50 seats; their average gain in a typical simulation run was between 47 and 48 seats. However, the playing field remains very broad and considerably larger are possible, as are considerably smaller ones."

Why would he not have added the huge margin of error +/- 30 at that time? The fact that he did not makes his original article very misleading and it is being spread all over the net.

http://fivethirtyeight.blogs.nytimes.com/2010/10/08/projected-republican-gains-approach-50-house-seats/

[tritsofme]

22. Right there in the first sentence he shows there is a 1/3 chance that Democrats hold the House

I'm really not sure what you are disputing, he explains his methodology in detail.

[Spazito]

27. His article does not state there is a +/- 30 margin of error...

and it is obvious he has found the need to clarify AFTER the article was published. Why did he not state that margin of error in his article in the first place instead of having to 'clarify' later. The margin of error was known at the time the original article was written but, rather than state it outright, it is left to the reader to figure it out.

A margin of error of +/- 30 renders his forecast useless, imo. He might as well have stated either the Dems or the Repubs will have control of the House after Nov. 2 election, it would have been more honest.

I have no bias against Mr. Silver and his forecasts, have read them with interest so this obfuscation of the margin of error, deliberate or otherwise, is quite surprising to me.

[monmouth]

14. So in other words, Nate knows shite....n/t

[ProudToBeBlueInRhody]

18. Nate is a right-wing Rethug M$M shill

Thought I'd say it, because it's only a metter of time before someone does.

[DemocratSinceBirth]

28. Larry Sabato Who Nailed 06 And 08

Larry Sabato who nailed 06 and 08 has the Republicans picking up 47 House seats, 7-8 Senate seats, and eight governorships.

[backscatter712]

29. The weatherman's conundrum.

Away from the studio and the green screen, the meteorologist makes his calculations, runs the statistical models using his satellite images, instrument readings, radar maps, etc. and gets a prediction that the temperature for today will be between 30 degrees and 80 degrees, using the usual margin of error.

So he splits the difference, and tells the people out in TV land that it'll be 55 degrees, and gets a whole bunch of complaints when it's actually 35 degrees.

The polls this year are all over the map. Anything indeed can happen.

[Aramchek]

32. what godd is it? well, it covers Nate's ass for one thing.

Whatever happens, he can claim to have predicted it.

But don't tell that to his apostles around here.

[Dawson Leery]

33. The one hit wonder is looking for another "hit".

By keeping the margins so wide, Nate can claim he was right all along. At the same time, we at DU understand the importance of GOTV.
That will determine who wins and who does not win this fall (as mid-term elections are determined by the partisan bases). We understand that
turnout will determine if we lose, 10 seats or 50 seats in the house. It will determine a loss of 2 senate seats or 10.
I am not putting too much stock into Mr. Silver. At the end of the day, he will be a one trick pony (2008).