Response to the Report Evaluation of Edison/Mitofsky Election System 2004

[KerryReallyWon]

National Election Data Archive Project Page 1 3/30/2005
US Count Votes'
National Election Data Archive Project
Response to the Report Evaluation of Edison/Mitofsky Election System 2004
http://exit-poll.net/election-night/EvaluationJan192005.pdf

Executive Summary
by Josh Mitteldorf

The most thorough and best-established exit poll for U.S. Presidential elections is the Edison/Mitofsky poll,
commissioned by major TV networks and print news services to predict the election outcome hours before
the official count is known. In November 2004, results of the poll differed sharply from the official vote
tally. In fact, the weighted national poll predicted a Kerry victory by 3% in the popular vote, while the
official count had Bush the winner by 2.5%1. This was the largest discrepancy in the poll’s history2.
The 2004 discrepancy arises in a context shaped by numerous reports of voting machine problems and
irregularities in the vote count, an overwhelming majority of which favored Bush. Can exit poll results be
interpreted as an indication of the global net impact of voting irregularities and bias in the official vote
count? Or is it more likely that a bias crept into the exit polls, and this error accounts for the bulk of the
discrepancy?
The Edison/Mitofsky organization argued for the latter hypothesis and warned against the use of their
polling data to discredit the official vote count. On 19 January, they released summaries and statistical
measures from their exit poll data. However, they have not released the raw data that would facilitate
independent corroboration of their analysis.
With the raw exit poll data, statisticians at E/M were in a position to extract several measures that could help
resolve the question whether the exit polls or the official count was at fault. The 19 January report includes
little of this kind of analysis; instead, almost all their analysis is predicated on the assumption that the
official vote count was accurate, and seeks to find fault with the polling methodology.3
So it remains for independent statisticians to ply the summary data which E/M have provided in pursuit of
the question: Was it more likely that the exit polls or the official vote count was at fault?
Our analysis of this question is based on summary data in the 19 January E/M report itself.
· Comparisons between exit poll results and official vote tallies by state.
· Corresponding comparisons for the 32 states in which Senate races were held.
· Table of precinct discrepancies according to partisan leaning of the precinct.
· Table of precinct discrepancies according to placement of the pollster.
· Table of poll cooperation rates according to partisan leaning of the precinct.
· Table of precinct discrepancies according to type of voting equipment.
More certainty would be possible if E/M were to release the full, precinct-level data set.
1 p. 20 Evaluation of Edison/Mitofsky Election System 2004 by Edison Media Research and Mitofsky International, Jan. 19, 2005
2 p. 34 Evaluation of Edison/Mitofsky Election System 2004 by Edison Media Research and Mitofsky International, Jan. 19, 2005
3 It is appropriate that E/M should focus their analysis on the factors under their control, and pursue exhaustively the possibilities
for error in their own work. To judge the integrity of the government’s electoral apparatus was never part of their mission, and
indeed to do so could affect their reputation and jeopardize their business future as America’s pollster of record.
National Election Data Archive Project Page 2 3/30/2005
Our conclusion is that the data appear to be more consistent with the hypothesis of bias in the official count,
rather than bias in the exit poll sampling. No data in the report supports the E/M hypothesis that Kerry
voters were more likely than Bush voters to cooperate with pollsters and, in fact, there is some indication
that the opposite may have been the case.
Our report is organized around three possibilities concerning the source of the discrepancy.
Was sampling error a significant factor?
E/M’s conclusion is that sampling error cannot explain the discrepancy between the exit poll results and the
official tally. Their reasoning is sound. With 70,000 respondents nationwide, the poll was designed to have
a margin of error under 1%. Computation of the theoretical uncertainty in the count is complicated by the
fact that the sampling was clustered at a subset of precincts that were hand-picked to produce a
representative sample of each state. But we agree with E/M’s conclusion that the observed discrepancy of
5.5% in the national survey was far outside any reasonable estimate of the poll’s uncertainty.
Seven of fifty* states have t values less than –2.7, meaning that each of their discrepancies had less than 1%
probability of occurring by chance. The binomial probability that 7 of 50 should be so skewed is less than 1
in 10 million. A full comparison of the exit polls with the null distribution4 that discrepancies are
distributed randomly by chance (blue curve) yields a probability that is astronomically small.
In the above plot, each green block represents one state. Many
more states are shifted to the left than to the right of the expected
distribution (blue line), but the distribution is not wider than
expected. The plot indicates that the difference between exit polls
and official results was due to bias rather than random error.
* The District of Columbia is in the database, but Oregon is not, because voting in Oregon is through US Mail, so no exit polls
were possible.
4 via a Shapiro-Wilks test
National Election Data Archive Project Page 3 3/30/2005
In the 19 January report, official vote tallies from the sampled precincts were substituted for exit polls in the
weighting formulas, to see if the results would correctly “predict” statewide voting patterns. This procedure
(pp. 28-30) confirms that the selected precincts accurately represented and predicted the election results in
their respective states, with only a small observed bias (0.3%) which was actually in the opposite direction
to the bias that resulted when exit poll numbers were used.
Was the exit poll biased by oversampling of Kerry supporters?
E/M demonstrate that the exit poll results from each precinct did not match the official tallies from these
same precincts. If the exit polls were at fault, there is therefore only one possibility: that Kerry voters were
more willing to fill out polling questionnaires than Bush voters. This is E/M’s hypothesis to explain the
discrepancy, offered prominently in their Executive Summary.
But statistical tests that could corroborate this hypothesis are conspicuously absent from the report’s body.
If Bush voters were indeed less likely to respond to the questionnaire, one would expect a negative
correlation between the response rate and the percentage of Bush voters in the precinct. E/M do not report
this correlation, but they do offer a summary table, in which the trend appears to go in the wrong direction.
The difference between the 56% response rate in Bush strongholds compared to the 53% response rate in
Kerry strongholds is marginally significant, based on the number and size of precincts that can be inferred
from data in the report. The dip at the plot’s middle confirms a recognized and expected trend: voters are
more reticent about making their opinions public in the presence of others with conflicting views. On this
basis, it would be expected that Bush voters are undersampled in Kerry strongholds, while Kerry voters are
undersampled in Bush strongholds.
In this context, if we attribute the discrepancy to Bush voters being shy about recording their views in the
presence of Kerry voters, then we expect the discrepancy should be lowest in Bush strongholds. Instead, we
see that the discrepancy was highest where Bush voters predominated.
National Election Data Archive Project Page 4 3/30/2005
Together, the trends in these two charts cast doubt on the primary hypothesis of E/M. In the body of our
report, we combine these two sets of results algebraically to show that the inconsistency is real, and that it is
severe. No plausible presumption about the rates of response among Bush and Kerry voters in various
environments can satisfy the conditions outlined in these two charts.
Although their hypothesis concerns bias in voter response rates, E/M do the responsible thing and
exhaustively explore the possibilities for errors, and the things they can do to improve their accuracy in
future polls, whether or not these are capable of explaining the observed bias. They document an increase in
the discrepancies with distance of the pollster from the polling place.
This data is useful for E/M in planning future polls, but it is almost certainly irrelevant to the 2004 exit poll
discrepancy. This is because the number of precincts where the pollsters were placed far from the polling
station was small. The discrepancies that E/M seek to explain are already fully present even in the precincts
where pollsters were optimally placed.
Most of this bias is apparent even in the 75% of precincts where the pollster was allowed to conduct his
survey just outside or within the building.
National Election Data Archive Project Page 5 3/30/2005
Do the exit poll results indicate bias in the official count?
The hypothesis that official vote counts are biased is not considered seriously in the 19 January report. The
only reason offered for dismissing vote count problems is that “in our exit poll sample overall, precincts
with touch screen and optical scan voting have essentially the same error rates as those using punch card
systems.” But this fact might also be construed as evidence that all four technologies have insufficient
safeguards to deter those who might be tempted to alter the equipment for partisan advantage.
Also consistent with the hypothesis of corruption in the vote is the finding that the mean discrepancy is
highest in Bush strongholds. It is easy to imagine that in precincts dominated by Bush supporters, the
temptation is greater to distort the count for Bush advantage because the risk of detection and punishment
would be least. Only in the most strongly Bush precincts did the mean discrepancy depart significantly
from the median (10.0% vs 5.8%). If indeed there was corruption of vote counts among the Bush
strongholds, this statistic suggests that its distribution was far from uniform, and would be consistent with
large-scale vote count corruption in a small proportion of precincts. This hypothesis would be easy to check
if E/M were to release their raw polling data.
Conclusion
The many anecdotal reports of voting irregularities create a context in which the possibility that the overall
vote count was substantially corrupted must be taken seriously. The hypothesis that exit polls accurately
reflected the will of the people and the official tally is in error is a theory with some support in the
Edison/Mitofsky data, while the same data casts doubt on the opposing hypothesis (that more Kerry
supporters than Bush supporters cooperated with the pollsters). A thorough investigation and careful,
exhaustive recount in key states would be an appropriate response.
For further information: contact Bruce O’Dell, Vice President, US Count Votes [e-mail]
612-309-1330 or visit www.uscountvotes.org
US Count Votes is a Utah non-profit corporation. Its goal is to provide nationwide, impartial statistical auditing services to
help ensure the accuracy of future elections. US Count Votes is seeking support for its "National Election Data Archive"
project in order to collect detailed election data and, prior to November 2006, develop statistical methods to audit elections
results data and provide statistical evidence of vote tabulation errors immediately following any U.S. election.

[librechik]

1. somebody needs to MAKE Edison Miofsky release the raw data

subpoena or law suit or whistleblower, the fate of our democracy depends on citizens getting that data away from the corporate entity that's holding it.

[jasmeel]

2. Is the Kerry campaign looking into it at all? n/t