The probability that this was going to be controversial was probably high.

Recently, The New York Times and CBS News began using an online methodology for their polling ahead of the 2014 midterm elections, relying on responses from more than 100,000 people nationwide gathered by online research firm YouGov.

The panel survey approach the publications used is somewhat controversial in the polling space because the sample doesn’t rely on probability. Anticipating this, blogger Nate Cohn highlighted the methodology used by YouGov, as well as the issues with traditional telephone-based surveys and the overall pros and cons of YouGov’s methodology.

“As the young voters who are less likely to respond to telephone surveys become an ever-greater share of the population over time, it is probably more important for analysts to have an ensemble of surveys using diverse sampling and weighting practices,” Cohn wrote.

The new surveying approach highlights a tough fact of life for the polling space. With the rise of smartphones, people use landlines less these days, and that’s leading to questions as to how the industry should adapt. While YouGov’s strategy is common in the world of marketing, it’s less so in the world of opinion-based forecasting—and that’s the crux of the problem.

The Concerns Raised

Soon after the poll came out, two major voices in the polling space—the American Association for Public Opinion Research (AAPOR) and Pew Research Center—raised serious questions.

In a letter, AAPOR President Michael Link, Ph.D., suggested that the approach used by the media outlets had “little grounding in theory” and noted the lack of transparency offered.

“Unfortunately, due perhaps in part to the novelty of the approach used, many of the details required to honestly assess the methodology remain undisclosed,” Link wrote [PDF]. “This may be an isolated incident with the Times / CBS News providing more information on this effort in the coming weeks. If not, it is a disappointing precedent being set by two of our leading media institutions.”

In an interview, Pew Director of Survey Research Scott Keeter said the test “is a very big deal in the survey world” and would likely encourage more experimentation, but noted a couple of significant concerns—mainly that there is a portion of the population not on the internet and that a replacement must show a solid track record before it’s adopted widely.

“The troubles faced by one method of research cannot, by themselves, be used to justify the adoption of an alternative,” Keeter explained.

Defending the Strategy

For their part, the two news outlets defend their track record and methodology.

“The New York Times and CBS News are proud to continue our long history of presenting new and methodologically rigorous analyses to our readers and viewers,” the news outlets said in a statement to Politico. “Our findings and methods for the new Battleground Tracker were laid out in great detail for all to see, and the underlying methodology is already well-known to, and widely used in, the survey research and scholarly communities. Moreover, both The New York Times and CBS News have successfully used online panel data in past projects.”

In the wake of the new polling approach, The Times updated its polling standards document, opening the door for new polling strategies such as the YouGov collaboration.

“The world of polling is currently in the midst of significant change, and The Times has begun a process to review its polling standards,” the document stated. “While the process is ongoing, the paper will be making individual decisions about which polls meet Times standards and specifically how they should be used.”

(photo by Daniel Oines/Flickr)