Column: Polls are right most of the time. This is what you have to be careful of

Without surveys you cannot understand politics; with them, you can misunderstand many things.

With the presidential nominations of both major parties secured, we are in a season when worrying about the polls can become an obsession. This is especially true this year, as former President Trump maintains a small but persistent lead over President Biden in most national and swing state polls.

That has led many Democrats to dig deep into the bowels of the polls in an often self-deceptive search for errors.

The fact is that polls continue to show correct election results the vast majority of the time. They are also an indispensable tool for democracy: informing the residents of a vast and varied nation what their fellow Americans believe.

At the same time, errors exist, often involving problems in collecting data or interpreting it.

This week, let's examine a couple of examples and take a look at how the LA Times polls fared this primary season.

A Holocaust myth?

In December, The Economist published a surprising poll: “One in Five Young Americans Thinks the Holocaust is a Myth,” read the headline.

Fortunately for the country, although perhaps not for the publication, it is the result of the survey that may have been mythical.

In January, the nonpartisan Pew Research Center set out to see if it could replicate the finding. They could not. Pew asked the same question as The Economist survey and found that the share of Americans ages 18 to 29 who said the Holocaust was a myth was not 20%, but 3%.

What is happening?

The problem is not a bad pollster: YouGov, which conducts polls for The Economist, is among the most respected polling organizations in the country. But the methodology YouGov uses, known in the survey world as participation panels, can fall victim to fake respondents. Maybe that was the case here.

Panel surveys are one way to solve a big problem that pollsters face: today, very few people answer phone calls from unknown numbers, making traditional telephone surveys extremely difficult to conduct and very expensive.

Instead of calling random phone numbers, survey organizations can solicit thousands of people to agree to take surveys, usually in exchange for a small payment. For each survey, interviewers select people from the panel to form a sample that is representative of the general population.

Some people join simply for money, however, they can then speed up the process and answer more or less random questions. Previous research by Pew has found that these fake respondents often claim to belong to groups that are difficult to recruit, including young people and Latino voters.

Pollsters have found evidence of organized efforts to infiltrate panels, sometimes involving “multiple registrations from people who are outside the United States,” Douglas Rivers, YouGov's chief scientist and political science professor at Stanford, wrote in an email. They could be efforts to boost particular causes or candidates or, more often, schemes to make money by raising small sums over and over again.

“We have ample procedures to rule out these panelists,” Rivers wrote, adding that the firm was continuing to look at what happened to the Holocaust issue.

In polls of close elections, fake respondents who respond at random typically “more or less cancel each other out,” said Andrew Mercer, a senior research methodologist at Pew.

“But for something that is very rare, like Holocaust denial,” random answers will produce an error that is entirely on one side. “This will end up inflating the incidence,” she said.

In previous research, for example, Pew found that 12% of respondents to voluntary survey panels who said they were under 30 also said they were licensed to operate a nuclear submarine.

The lesson here is an old one, popularized by the late astronomer Carl Sagan: “Extraordinary claims require extraordinary evidence.” If a survey result seems too surprising to be true, it probably isn't.

Jumping to conclusions

A second category of potential problems has less to do with the data than with the way people, especially us journalists, interpret it: drawing definitive conclusions from figures that are not definitive.

Let's consider the question of how much progress Republicans are making among black and Latino voters.

There is no doubt, as I wrote previously, that Republicans gained ground between 2016 and 2020, especially among Latino voters who already identified as conservative. There was also less movement toward the Republican Party among black voters.

Has that trend continued? Some recent polls, including the widely cited New York Times/Siena College poll, indicate that it may have accelerated. According to that poll, Biden is hemorrhaging support among younger black and Latino voters.

In a recent article that attracted a lot of attention, John Burn-Murdoch, the chief data journalist at the Financial Times, pooled data from several different types of polls to declare that “American politics is in the midst of a racial realignment.”

The response from many political scientists and other analysts was, indeed, “not so fast.”

Pre-election polls can indicate what potential voters think today, but comparing them to the results of past elections is risky, they said.

If the actual results in 2024 follow what the New York Times and Siena polls are currently finding, “well, let's talk about racial realignment,” said Vanderbilt University political science professor John Sides. Until then, however, “we have to wait and see.”

How did we do it

Our UC Berkeley Institute of Governmental Studies/Los Angeles Times polls had a remarkably good year for election prediction.

The final poll before this year's primary showed, for example, that Proposition 1, the $6.4 billion mental health bond measure backed by Gov. Gavin Newsom, had the support of 50% of voters. likely voters.

As of Thursday morning, that was almost exactly where the “yes” vote stood (50.2%) with nearly 90% of the state's votes counted.

The poll also correctly predicted that Democratic Rep. Adam B. Schiff of Burbank and former Republican Dodgers player Steve Garvey would be the top two finishers in the Senate primary, with Democratic Rep. Katie Porter of Irvine in third place.

In the poll, conducted about a week before the election, 9% of voters remained undecided. Among those who had made up their minds, Garvey got 30% of the vote, Schiff 27% and Porter 21%, according to the poll.

The poll appears to have been very close to Garvey's figure: with about 800,000 votes still to be counted, he has 32%, well within the poll's estimated margin of error of 2 percentage points in either direction. The poll slightly underestimated support for Schiff, who also has 32%, and overstated support for Porter, which currently stands at 15%. That could mean the last group of undecided voters opted for Schiff.

That level of precision is not uncommon. In the 2022 midterm elections, for example, polls conducted by nonpartisan groups, universities, and media organizations were extremely accurate.

There's a takeaway in all of this for people interested in politics, especially in a hotly contested election year: Don't focus too much on any single poll, especially if it has a surprising finding that hasn't emerged anywhere else. Be skeptical about sweeping conclusions about events that are still unfolding. And even, or perhaps especially, when a poll shows your favorite candidate trailing, take it for what it is: neither an oracle nor a nefarious plot, but a snapshot in time.