When President Obama announced that he would delay executive action to protect some undocumented immigrants from deportation until after the midterms, it was widely recognized as a political move intended to shield Democrats from backlash. The logic never made much sense—certainly not from a moral perspective, but neither from a political one. It wasn’t clear how delaying could actually improve already grim circumstances for vulnerable incumbents. Meanwhile, it was obvious that letting mass deportations continue on a purely political whim would alienate many Latino voters.

Well, that’s exactly how it played out. There’s no way to know if things would have been even uglier for the left had Obama issued the orders. But what the midterms did suggest is that the Democratic lock on the Latino vote has slipped. Not only the White House’s delay but also reluctance to embrace a pro-immigrant platform cost Democrats votes in key states.

Nowhere were Latino voters more important than in Colorado, where Mark Udall lost to Cory Gardner. Gardner knew it, and he ran as an ally to the immigrant community, even though he opposed the Dream Act and comprehensive immigration reform. Udall, despite his strong record of support for immigrant rights, never drew a stark distinction between himself and Gardner on the issue.

The result, according to figures released Wednesday by the polling group Latino Decisions, was that voters were confused about where the candidates stood on the issue—which more than two-thirds of Colorado’s Latinos identified as a priority at the polls. Forty-seven percent said they weren’t sure whether Udall supported immigration reform; 6 percent thought he opposed it. Gardner, meanwhile, got away with his bait-and-switch: only 38 percent knew he opposed reform.

Udall still won the Latino vote handily, 71 percent to 23 percent. However, that gap doesn’t look so great compared to other recent elections. Barack Obama pulled in 87 percent of Colorado’s Latino voters in 2012, and Democratic Senator Michael Bennet won with 80 percent of the vote in 2010. Turnout wasn’t bad in Colorado; it just wasn’t as deeply blue.

In fact, there was a dramatic decline in Latinos’ support for Democratic candidates in Senate, House and gubernatorial races all over the country. Kay Hagan, who lost her Senate seat in North Carolina to Tom Tillis, captured only 63 percent of the Latino vote, nine points below the support Obama had in 2012. Hagan ran to right on immigration; she opposed the Dream Act and executive action on deportation. It’s not clear whether her hardline stance won her any votes—certainly not enough to make a difference. What it did do was focus negative attention on her from the immigrant-rights community. After the White House delayed executive action, Hagan (along with several other Democratic candidates who’d asked for it) came under fire from groups like Presente, which ran ads in North Carolina that attacked her for siding “with the most anti-immigrant senators in Congress in support of the continued deportations of our community.”

Source: Latino Decisions

One of the most shocking shifts occurred in Nevada, where Republican Governor Brian Sandoval swept to victory with 47 percent of the Latino vote. In 2010, 84 percent of Nevada’s Latino voters picked the Democratic candidate. Sandoval’s improvement is likely due to the fact that he’s taken real steps to pass policies important to the Latino community, including expanding the state’s Medicaid program, signing a law permitting undocumented immigrants to obtain drivers licenses, and increasing financial support for students learning English in school.

The lesson is that Democrats can’t take Latinos for granted, said Gary Segura, co-founder of Latino Decisions. “Latinos remain a swing electorate,” he said at the National Press Club in Washington on Wednesday. “Republicans can have access to Latino votes. They just need to try, by producing policies that are of interest to Latino families and communities.” He said that Latino voters “clearly feel ignored by the Democrats,” even if they also feel like they are under direct attack from the GOP.

It was turnout, not a party shift, that most people worried about ahead of the midterms, and indeed Latino participation was two points lower nationally than in 2010. Latino Decisions found some evidence that this was related to the delay in executive action. In a sample of Latinos who were registered to vote but chose not to, 60 percent said it made them less enthusiastic about supporting the Democratic Party. “We have, for the first time, some evidence to suggest that for some of these voters who didn’t vote, in fact, they were mentioning [the delay] as a reason for the fact that they were less enthusiastic about Tuesday, November 4^th,” said Latino Decision’s other co-founder, Matt Berreto.

“I believe this election was a lost opportunity for Democrats,” said Janet Murguía, president and CEO of the National Council of La Raza, one of the sponsors of the poll. “Instead of embracing their record and our community, too many Senate Democrats either ignored or ran away from us.” She called the Democrats voter outreach and mobilization in Hispanic communities “anemic.”

The gap between Udall and Gardner was large enough that even a stronger show of support from the Latino community might not have allowed Udall to fully close it. Similarly, Democrats might not have been able to overcome the host of challenges the midterm map presented even with higher levels of enthusiasm among Latinos. But it’s clear that walking away from immigration made a tough night worse for Democrats—and that allowing Republicans to define the conversation around immigration (or the Keystone pipeline, or Ebola, etc.) in contested states is not a winning position.

On Wednesday, Obama reiterated his promise to use his executive authority to protect some immigrants from deportation. Advocates are already urging him to go big, and warning about the consequences for Democrats if he doesn’t. “This problem that you see, politically, is nothing in comparison to the civil war that will be created politically in the Democratic party should the president not be broad and generous in his use of prosecutorial discretion,” Illinois Representative Luis Gutiérrez said in an interview with The Guardian ahead of the midterms. “Because Latinos will not be deciding whether or not they vote, but whether or not they are in the Democratic Party.”

Republicans will freak out no matter the scope of Obama’s actions, so he might as well do something that really protects families and eliminates heavy-handed enforcement tactics like the dragnet-style raids in Louisiana that I’ve reported on. Unfortunately, the president has often expressed unfounded optimism that limiting his hand will provoke a similarly measured response from the right. (Over at Vox, Matt Yglesias explains why he thinks the Republican takeover dims the chances for a substantive directive from the White House on deportation.) The question now is whether commitment to the optics of cooperation—which blurs sometimes with appeasement—will trump the human calls for relief.