Polling the Pollsters: Are Surveys Becoming Unreliable?

Polling professionals say that fewer people are responding to surveys, which dilutes the credibility of the results. What can associations do to keep their survey findings reliable? The leader of a researchers’ organization shares some tips.

With the midterm elections less than a month away, media coverage of the latest political polls has ratcheted up once again. The website—run by Nate Silver, who made his name by correctly predicting the outcomes of the 2012 presidential election in 49 of 50 states—dedicates a large chunk of its coverage to these polls and their accuracy.

The quality of the results “really gets down to the level of understanding and training of the people who are conducting the poll.”

But Carl Bialik, the site’s lead writer for news, decided to get a little meta this week: He polled the pollsters about, among other things, the work that they do. Bialik reported that the average election poll response rate in 2014 was 11.8 percent, down almost two points from 2012.

That doesn’t sound like much, but the drop is consistent with a larger trend that the polling industry has seen over the last few decades: People are becoming increasingly difficult to reach, and it’s having an impact on the accuracy of polls and surveys across the board.

“These same trends are affecting all types of polls, whether they’re done by the government, done for the healthcare industry, market research, JD Powers—it’s the same phenomenon that everyone is facing,” said Michael Link, president of the American Association for Public Opinion Research. “There’s any number of reasons for this. Some of it is technology, and some of it has to do with a basic shift in people’s behavior.”

Through studies, AAPOR—which makes a variety of polling resources publicly available—has discovered that the declining response rate can correlate to less-reliable surveys, but the organization has also done research to identify techniques and strategies to counteract it. Link offered three tips for executing a successful survey:

  1. Sampling. You need the right mix of people from your industry or membership. “If you can’t interview everyone, you need to draw a sample, so how do you draw a sample that best reflects the industry? Is it by company size, by types, by some combination thereof?”
  2. Format. Determine the best method for conducting the survey. “Today, most professionals can be reached through some form of online poll,” Link said. “But maybe you’re dealing with an older generation of folks, so online might not be the way to go.”
  3. Make adjustments. If you don’t get the response rate that you were looking for, all is not lost, Link said. Have people on the back end who can “look at the raw data after it is collected to ensure that the characteristics of those who respond are similar to those of the broader group being surveyed.” If needed, Link said, researchers can use accepted statistical techniques to weight the responses to make the data more representative of your membership.

He added that a survey’s accuracy can have a lot to do with the people who conduct it, noting that the ease of polling with online tools has made it easier for almost anyone to deploy a survey, even without training in research best practices. The quality of the results “really gets down to the level of understanding and training of the people who are conducting the poll,” Link said. “Ensure that you have someone who is trained in conducting surveys leading the effort or just as a part of the research team.”

Do you have survey tips to share? Leave them in the comments.


Rob Stott

By Rob Stott

Rob Stott is a contributing editor for Associations Now. MORE

Got an article tip for us? Contact us and let us know!