Cautionary article on polling data from a social scientist:

[Emillereid]

September 20, 2004


David Price
Relying on Phonies: What If The Problem with Phone Polls is That They Are Phone Polls

What If the Problem with Phone Polls is that They are Phone Polls?


By DAVID PRICE


<snip>



As a social scientist who uses both quantitative and qualitative research methods, I am not one to dismiss survey research out of hand, but something has methodologically gone awry when polls are swinging about this wildly. Obviously past telephone-survey presidential polls have accurately predicted election outcomes, but Americans' social interactions via telephones may be evolving in ways that render past telephonic sampling techniques unreliable.


We Americans simply don't answer our phones like we used to. Entire industries are now devoted to helping us not answer the phone. Voicemail, Caller ID, caller-specific-rings, cell-phones, even email have fundamentally transformed the ways we (don't) answer the phone when it rings. These and other technological innovations have moved us from a late-20th Century near-pavlovian automatic response of answering the phone when it rang, to new levels of screening or ignoring calls without a sense that we might be missing something important. When pollsters call under these technological conditions they are now increasingly treated as any telemarketer or unknown caller would be, thus the people who pollsters actually get to talk to are becoming increasingly less representative of the general public. There now may be something unusual about people who are willing to answer the phone a talk with strangers, and we should be skeptical about generalizing from the results of these surveys. It is possible that the new habit of non-phone-answering is evenly distributed throughout the population (thus reducing this as a sampling confound), but this seems unlikely.
<snip>

"More African-Americans than Whites have caller-ID (73% vs 47%) and a higher percentage of Blacks use it for call screening (34% vs 24%). Young people, ages 18-29 are the group most likely to say they always screen calls with caller-ID (41% say this), compared with only 12% those aged 65 or older."


Pew also found that more women than men were found to use features like call blocking (20% vs. 14%). If we can get over the paradoxical fact that this data was collected in phone interviews (and of course the point of this piece is that I'm not sure we can get over that) you can see that those profiled as being most prone to answering phone surveys tend to be: (more) White, (more) older, and (more) male. Or if you prefer to think this through in hall-of-mirrors-phone-paradoxical-mode: We simply don't know how many households with Caller-ID were called and didn't choose to answer. Out of those homes who did answer the phone it was reported that those who didn't use call screening were more white, male & older. But for all we know there is a whole universe of households with the opposite attributes who used Caller-ID to avoid this poll.

more at: http://counterpunch.org/price09202004.html



This begins to answer the wildly fluctuating polling data -- although the way I was taught statistics one should just keep calling until all the representative sub- populations have been sampled -- don't they do polls that way anymore. Do they just run with the data knowing they haven't enough from some given population, say women or younger people or people of color or people who live in different geographical samples or .... If so, these data are indeed bogus!

[PittLib]

1. I do not answer

"unknown"s. I am an unlikely poll candidate.

[delete_bush]

2. This could be good news for us, sounds

like the ones who answer pollsters are more typically repubes. This could be the reason that some of the recent polls I've read tend to sample more R's than D's.

[izzie]

3. That is interesting. I have none of those service but often don't ans.

You know I hear people say they have never been polled but I have, and in this way. Phone where they said who they were and told me who did the poll and how long it would take. One other just asking if I voted and who would I vote for. once just the same questions on street and once if I would take a written poll in a mall, where I had to sit and fill out a few pages of questions. I recall them as I have always had to work to vote. I was married to a service man and had to do my own work to get the forms to vote and it was drilled into as a child that my mother was born when she could not vote so I better do it. I always have.

[pacalo]

4. Great article! I don't answer "out of area" phone calls, either.

[secular_warrior]

5. This far out, polls aren't accurate

and the way we as a society use phones probably has something to do with it. Liberals and other Democratic constituencies (young professionals, women, minorities, working poor) tend to lead faster paced urban lifestyles (most of the Dem vote comes from metropolitan areas) and would likely not be home to answer the phone, and even if they were, would be too busy/tired to answer it.

My theory about polls is that people start answering these calls about two weeks out from the election as they become engaged in the election, resulting in a more accurate sample. This is the only explanation for the dramatic difference in polls during the dog days of the campaign and the weeks before an election... undecideds and switchers are not enough to explain it. We saw this in the Democratic Primaries, when all of the polls showed Dean as the inevitable winner with a commanding lead. That race did not change dramatically in the final week, IMO, as much as the polls only started to accurately reflect true voter sentiment. I believe this will happen again, and the polls will turn sharply against Bush in the final two weeks of this campaign. I have nothing but my own instinct to judge it on, but I trust that Americans will not re-elect this failed president, regardless of what the media is saying. Remember, the media was crazy about Dean and ignored Kerry, and justified their focus on Dean by the citing his high poll numbers, when Dean's support all along was way behind Kerry's, IMO.

[AndyTiedye]

6. Maybe the Pollsters Shouldn't be so Damned Anonymous

If the caller-id says "unknown", we let the answering machine screen it.
but if it said "HARRIS POLL" or some other poll, even "GALLUP POLL",
I'd answer it.

[Tansy_Gold]

7. I don't buy this.

Emillereid wrote:

"This begins to answer the wildly fluctuating polling data -- although the way I was taught statistics one should just keep calling until all the representative sub- populations have been sampled -- don't they do polls that way anymore."

Personally, I think they DO still do polls that way -- filling the desired groups and subgroups in a predetermined sample, which would effectively negate the effect of changing telephone habits.

But I also think they're skewing the sampling mix. There are a lot of ways to do it, and I think the changes in sampling effect the swings more than anything else.

However, the polls GENERALLY focus on "likely" voters, which means those people who have been registered for more than one or two election cycles and have voted in most of them. That can leave out a lot of 18-24 year olds, who have traditionally not been either registered or voting, who have not had as much opportunity to vote, and are a sizeable portion of the potential electorate. Of course, if they don't get out and actually cast ballots in this election, they won't have any effect on the outcome any more than they would have in the polls.