WASHINGTON – As the election nears, not only candidates face a reckoning. Pollsters, too, confront a moment of truth. The close election could leave many calling the wrong winner. On a recent day, six national polls reported new results. Four had Mitt Romney leading; two had Barack Obama. The largest victory margin was 3 points (for Romney). The average margin was 2 points. The uncertainty has fueled speculation that Romney could win the popular vote and lose the electoral count.
It’s not just the election. Among pollsters, there’s fear that changing technology (mainly cellphones) and growing public unwillingness to do interviews are undermining telephone surveys – and that there’s no accurate replacement in sight. A recent study by the Pew Research Center reported its response rate at 9 percent, down from 36 percent in 1997. Put differently: in 1997, Pew made about three residential calls to get one response; now it makes 10.
Beginning with answering machines and caller-ID in the ’70s and ’80s, suspicious Americans have become more selective in screening calls. Robo calls – automated messages for products, politicians, charities and polls – have deepened the hostility. “The mass of communications coming into people’s homes ends up being a blur,” says Pew pollster Scott Keeter.
Cellphones pose problems because people who use them exclusively – people who don’t have landline phones – are younger, poorer and more Democratic than the general population. By late 2011, 32 percent of Americans 18 and over had only a cellphone, up from 16 percent in early 2008. Among those 25 to 29, the share was 60 percent. Under-surveying these people could distort polls. Many pollsters, though not all, now canvass cellphones. But this is increasingly expensive. By present trends, half of Americans could be exclusive cellphone users by the 2016 election.
All this threatens the largest upheaval in polling since the 1930s. Until then, Americans gauged public sentiment through newspaper straw polls, “speeches, petitions, rallies, riots, strikes, elections and letters to the editor,” writes historian Sarah Igo of Vanderbilt University in her book, “The Averaged American: Surveys, Citizens, and the Making of a Mass Public.”
The big change came in 1936, when George Gallup and Elmo Roper, using population sampling techniques, forecast Franklin Roosevelt’s victory. Meanwhile, The Literary Digest magazine, relying on a straw poll of its readers, predicted Republican Alf Landon. The magazine’s readers were unrepresentative of the whole population. Scientific sampling, which attempted to reflect everyone, triumphed, though it suffered a temporary setback in 1948 when polls missed the late surge toward President Harry Truman.
Gallup and Roper were to some extent idealists, says Igo. They thought that modern polling would improve democracy by clarifying the majority’s desires. It hasn’t worked out that way, in part because polls, by exposing sizable minorities, also strengthened their sense of identity.
But polls did change how we talk and think about ourselves. Concepts such as “the typical American,” “mainstream culture” and “public opinion” became common, writes Igo.
Polls are controversial today because the presidential race is tight. If either candidate had a 10-point lead, few would care how different polls decide who’s a “likely” voter or how various population segments are weighted. These details usually don’t matter much. But when the margin is a few points, they can determine which candidate is the front-runner and has “momentum.”
More menacing is whether cellphones and shrinking popular participation subvert polls’ accuracy, because samples are less representative. The answer seems to be: “not yet.” Pew did a study comparing responses to basic questions (are you: U.S. citizen, homeowner, married?) from its surveys to larger government surveys.
On most questions, responses were identical or close: 37 percent of respondents in both had children in the house; 75 percent in both were registered to vote. There were a few troubling discrepancies – but no fundamental break.
Less reassuring is telephone polling’s steep and rising costs, which could cause cash-strapped media organizations to balk. Contacting cellphones is expensive, because numbers must be dialed by hand. By contrast, computers can automatically dial landline numbers, making it easier to reach live people. (Congress prohibited this for cellphones to protect people from paying for unsolicited incoming calls.) A typical survey costs Pew from $60,000 to $100,000, says Keeter. That would cover renting tens of thousands of landline and cellphone numbers to produce 1,500 interviews of about 20 minutes each.
The solution seems obvious: switch to the Internet. But technically, that’s hard. Internet users may not be a representative sample of the U.S. population. Does the person behind that email live in the United States? Permanent panels of respondents may act differently from randomly contacted people.
Experiments are under way. Meanwhile, pollsters are stretched between a past that’s growing untenable and a future that doesn’t yet exist.
Robert J. Samuelson is a Washington Post columnist.
JOIN THE DISCUSSION | Register here
We welcome comments. Please keep them civil, short and to the point. ALL CAPS, spam, obscene, profane, abusive and off topic comments will be deleted. Repeat offenders will be blocked. Thanks for taking part — and abiding by these simple rules. A thorough explanation of rules of conduct can be found in our Terms of Service. If you have any questions, including why your comment may not be showing immediately after you submit it, be sure to visit the commenting FAQ.