It’s said to be wrong to kick a person when he or she is down. If Monday’s New York Times/Siena poll were a person, it’s been stomped so severely that a compassionate observer would step in to stop the fight. But even though the poll that launched a thousand headlines claiming the midterms are moving back toward Republicans, and that the so-called Dobbs effect—a shift to Democrats after the Supreme Court did away with a 50-year-old constitutional right to abortion—is subsiding, has been pretty thoroughly debunked by pollsters and progressive analysts, it still deserves attention (but no kicking here, folks).

It’s a case study of what even “good” polls can do wrong, and, maybe more important, of how journalists looking for a “new” story line hype outlier polls without understanding the first thing about what they mean—as well as the way voters should think about new polling as we get closer to the crucial election.

In case you were without a computer or television earlier this week, here’s the gist of the poll of 792 “likely voters.” In September, those polled by New York Times/Siena favored Democrats on a “generic” congressional ballot, by one point. A month later, those polled back Republicans by four. The big news, from the Times headline: “With elections next month, independents, especially women, are swinging to the G.O.P. despite Democrats’ focus on abortion rights.” The economy, the poll found, mattered much more to voters than abortion.

And despite the fact that a “gender gap” showing women favoring Democrats has been a defining feature of American politics since Ronald Reagan’s election in 1980, the poll showed women dividing their votes equally between the two parties. “The poll showed that Republicans had entirely erased what had been an 11-point edge for Democrats among women last month in 2022 congressional races to a statistical tie in October,” the Times wrote.

The detail that got the most hype, though, from the Times write-up: “The biggest shift came from women who identified as independent voters. In September, they favored Democrats by 14 points. Now, independent women backed Republicans by 18 points—a striking swing given the polarization of the American electorate and how intensely Democrats have focused on that group and on the threat Republicans pose to abortion rights.”

Wow. That’s a 32-point swing. Big if true.

But there is no reason to believe it’s true.

Let me stop here and say: It’s entirely possible, maybe likely, that momentum has swung in Republicans’ favor over the last month. The Democrats’ summertime wins—a gun safety bill, the Inflation Reduction Act (and especially its provisions cutting Medicare costs for seniors), declining gas prices, and the Kansas victory on abortion—may have faded in memory. Persistent inflation, gas prices’ rising again, the declining stock market, and other forms of economic pain are genuine concerns among voters. “We may be returning to a more traditional model,” where the president’s party almost always loses congressional seats in its first midterm election, admits Democratic pollster Celinda Lake, who is nonetheless a fierce critic of the latest Times/Siena poll.

TargetSmart CEO Tom Bonier, another poll critic, agrees. “There’s a plausible case to be made in this election that the GOP will do well,” Bonier says. But the Times/Siena poll doesn’t make it. “I break down its flaws to remind us: Never pay attention to a single poll. View these polls as a ‘what if’ scenario,” Bonier says. “They’ve constructed a ‘likely voter model,’ with a potential outcome that’s plausible.”

But that “likely voter model” is problematic in this election, some pollsters say. “We can’t build a likely voter model that says ‘Let’s go look at the last time half the population lost a fundamental right,’” Bonier notes.

The biggest flaw in the poll, which was sadly the fact most hyped by mainstream journalists, was that alleged 32-point swing among “independent women” to Republicans. It’s based on 95 women, and its margin of error is at least 10 points. “Nobody should have reported that as truth,” Bonier told me.

Also, many journalists don’t seem to understand what’s behind “likely voter” models. Even strong polling organizations make choices about which demographic groups are likely to turn out, and in what numbers, based on shifting political winds. To polling newbies: That means the Times/Siena did not poll the identical 792 voters in September and October to find those disturbing swings towards Republicans. Good pollsters, and Times/Siena is considered one of the best, have to make judgments each time they poll about who’s “likely” to vote. But that means month-to-month “swings” in voter preference like the Times-reported September Democratic advantage that tilted towards Republicans in October represent apples-to-oranges comparisons.

It’s true that many pollsters overestimated Democratic margins in 2020, and even predicted wins that turned to losses in 2016 (sorry, President Hillary Clinton). It’s widely accepted that they somehow missed a lot of Republican voters, whether because they didn’t reach them or their “likely voter” models were skewed. “They don’t want to have a Democratic bias in their polls/likely voter models this year,” Bonier says, “so it’s possible they’re overcorrecting with Republicans.”

If pollsters were transparent, he adds, they’d share their “likely voter” models, and even release different scenarios based on different turnout assumptions. “But pollsters are seen as gurus—they can’t be wrong. If they admit their polling is based on these ‘models,’ does that mean they don’t know what’s going on?”

Bonier, Lake, and other pollsters agree: The decisive “tell” that the poll was flawed was its finding that women are splitting their votes evenly between Republicans and Democrats. “Do you really believe just months after losing a fundamental right, women will split their votes [between Republicans and Democrats]?” Bonier asks. “Have we ever since the ’90s had a situation where women didn’t vote more Democratic than men did?” pollster Anna Greenberg asked rhetorically in The New Republic.

Lake was more scathing: “There isn’t another poll in America that shows that,” she says. “If I did an outlier poll like that for a candidate, I’d have to do it over again at my own expense.” The Times should have tossed its October findings and started over, she says.

The best “polls” are of course actual elections, and Democrats have outperformed expectations in most of them this summer, thanks largely to increased turnout among women and young voters. In the special election for New York’s 19th Congressional District in August, there was a seven-point gender gap favoring Democrat Pat Ryan; Joe Biden’s edge among women in 2020 was only four and a half points. “I’m not aware of a single poll in that race that predicted a seven-point gender gap,” Bonier says. Voter registration is surging among women and young voters, he adds. That doesn’t translate to turnout, however, pollsters are quick to admit. Without targeted intervention, many newly registered voters may not show up in November.

Still, Bonier notes, “one scenario that’s not too far-fetched is that post-Dobbs we’re going to see higher turnout among women and younger voters,” especially given those registration numbers. But most “likely voter” models won’t capture that.

Since all reputable pollsters warn not to take any one poll seriously, why are so many people focused on refuting this one? The main reason is the way the media hyped it. The story lines—Women shift to Republicans! The Dobbs effect is dead!—dominated cable news and many online and print news outlets on Monday and Tuesday. “The danger is the extent to which the media takes a poll like this and turns it into a momentum narrative,” Bonier observes. “It doesn’t help [Democrats], not with fundraising. Can it dampen turnout? Potentially.”

One thing pollsters and advocates I talked to, as well as those canvassed by other curious journalists, agreed on: The Dobbs effect is not going away. But Democrats have to improve their messaging on the economy.

When Senator Bernie Sanders wrote in The Guardian that “Democrats shouldn’t focus solely on abortion. That’s a mistake,” he irritated a lot of feminists, including me. For one thing, he was setting up a straw woman: I’m not aware of one single candidate focused “solely” on abortion. “I think it bothered some of us because it seemed dismissive of how hard the movement has worked,” says Mini Timmeraju, president of NARAL Pro Choice America. But she doesn’t entirely disagree with Sanders. “We do have to make a broader economic case,” she says. Also, given the history of the president’s party suffering midterm losses, she adds, “the headwinds are still pretty strong against us. This is a lot to put on the abortion movement.”

Lake, who polls for pro-choice groups and Democratic candidates, as well as nonpartisan organizations like the AARP, says Democrats have to make their economic message as strong as their abortion message. “Why aren’t more people saying, ‘We capped insulin prices, we gave you hearing aids, we capped Medicare expenses, and Republicans will take all that away’?” the pollster asks. “Why aren’t they saying Republicans are promising to destroy Social Security and Medicare?”

The number-one group of swing voters, Lake says, “are women over 50, and these issues test off the charts with them.” The younger segment, women 50 to 60, are particularly incensed—once they learn about the Republicans’ plans, that is, and Democrats are not doing a good job at that. The older women assume they’ll keep their benefits (although nobody really knows), while the younger women “know they don’t have time to establish an alternative to Social Security before they retire.”

Still, her polling has found that abortion matters to these older women, too. “Some remember when it was illegal. Endangering miscarriage care is also very salient to them,” the pollster says. “And they don’t like the idea of taking away rights from their daughters and granddaughters. ‘We’re going backward? That’s not the way things are supposed to work.’”

Timmeraju says she’s encouraged by what she’s seeing on the ground. The NARAL president has visited five states in nine days, “and the energy is incredible. Volunteers, canvassing—we’re seeing presidential-cycle enthusiasm.” And while Lake urges more robust economic messaging, she also insists that continued strong abortion messaging is crucial. “Abortion is literally our best ‘get out the vote’ message—it’s how a lot of candidates will drive turnout. We need to see higher turnout by women and younger voters, and that’s a message that can help.”

Meanwhile, Timmeraju says she’s mostly ignoring polls at this point, less than three weeks before the election. “I am telling people on the ground they should not spend much time looking at the polls. Don’t freak out. Stay the course.” Words to live by for everyone at this point.