Democratic mayoral candidates, from left to right: Bill de Blasio, John Liu, Christine Quinn and Bill Thompson. (AP Images)

On election night in 2009, I ran into a prominent New York City progressive political operative. I asked him if he thought the Democrat running for mayor, William Thompson, stood any chance against two-term incumbent Michael Bloomberg. He gave me a dismissive smirk. “Come on,” he said. “Bloomberg’s got every political consultant in town. Who’s Bill Thompson got?” I named Thompson’s campaign manager. “Yeah,” the operative said. “Him against the world. Not a fair fight.”

Two hours later at Thompson’s headquarters, when early returns showed the Democrat running neck and neck with or even slightly ahead of Bloomberg, I saw the same progressive operative in a corner texting furiously in an apparent mixture of excitement and panic—seemingly both thrilled and terrified that New York might elect a candidate he agreed with but had counted out.

In the end, of course, Bloomberg won, and Thompson became the fifth consecutive Democrat to lose the mayoralty in a city where his party enjoys a six-to-one registration advantage.

Now New York is preparing to vote for mayor again, and polls suggest that Democrats have a powerful advantage. But in more than one recent mayoral race, progressives have seen early optimism vanish by Election Day. And a Democratic win doesn’t guarantee a progressive victory: the Democrat now leading the pack is Christine Quinn, who as speaker of the City Council largely cooperated with the centrist Bloomberg administration.

Thus, the 2013 race frames the same question that’s been asked after every mayoral campaign since 1993, and all but one since 1977: Why, in such a famously progressive place as New York City, is it so hard to elect a progressive mayor?

* * *

Gracie Mansion’s Ghosts

The old adage has it that there’s no Democratic or Republican way to take out the garbage. There is, however, a clear difference in how progressives and conservatives take out the trash—who should take it out, how much they should be compensated for it, where to send the garbage, how to get it there, and who should pay for the whole operation. Simply put, those who believe that a primary responsibility of government is to try to make the world a more just place—to help those with less power against those with a lot—are progressives, and New York City used to elect them as mayors.

Modern New York City politics begins in 1933 with the election of Fiorello La Guardia, a progressive Republican who built low-income housing, schools and hospitals and helped shape the urban aspects of the New Deal. He was elected to three terms. After two subsequent managerial mayors, Robert Wagner Jr.—far from a pure progressive, but a mayor who built housing for the poor and incorporated minorities into the civil service—served for twelve years.

Then came John Lindsay, a liberal Republican (he later switched parties) who sought racial harmony and better conditions for the poor, but took his hits for shaky management and alienating working-class whites. Lindsay left office widely regarded as a failure—an impression that hardened when budget practices linked to him were blamed for the city’s near bankruptcy. The fiscal crisis consumed Lindsay’s successor, Abraham Beame, dooming his 1977 re-election bid.

That ‘77 contest was a turning point in the city’s political narrative. Onetime left-winger Edward Koch campaigned in part on his support for the death penalty—irrelevant as a municipal matter but powerful as a symbol of his break from traditional progressive pieties. Koch won and governed for twelve years, much of that time spent in conflict with minorities and in cooperation with wealthy developers. He was ousted in the 1989 Democratic primary by David Dinkins, who then beat Republican Rudy Giuliani to become the city’s first black mayor.

Dinkins’s achievements as mayor were overshadowed by a crime wave and a nasty recession. Giuliani won the rematch in 1993, then introduced a menu of budget cuts, workfare and aggressive police tactics. He was easily re-elected in 1997.

In 2001, Bloomberg won a narrow contest over progressive Democrat Mark Green, who was hobbled by a bitter dispute within the Democratic Party about whether Green’s campaign had used racially divisive tactics to win the primary against the strongly progressive Fernando Ferrer. Four years later, Ferrer became the city’s first major-party Latino mayoral nominee but ended up being dramatically outspent by Bloomberg, who won by a landslide. After getting the City Council to overturn term limits, Bloomberg prevailed again in 2009.

Political leaders are complex creatures who both shape and are shaped by prevailing conditions, so even mayors who weren’t progressive have sometimes found common ground with the left. Koch built hundreds of thousands of units of affordable housing. Giuliani was fairly friendly to immigrants and gays. Bloomberg, a pro-business centrist, pursued gun control and an ambitious environmental agenda. Fiscal conservatives disparage all three for increases to the city budget.

But while Koch was complicated, Giuliani not uniformly reactionary and Bloomberg no conservative, none of them were progressives. At best, with a nod to Beame and Dinkins, one could say New York has had eight years of progressive rule in the past four decades. At worst, there’s an argument that the city hasn’t had a progressive leader since Lindsay left City Hall, reportedly with tears in his eyes, on the last day of 1973.

* * *

Demographic Destiny

To explain this trend, one has to figure out how much of the city’s recent past can be captured by a single narrative, and how much is the product of unique battles every four years.

One unifying truth, says University of Massachusetts professor and Lindsay biographer Vincent Cannato, is that New York City may never have been quite as liberal as people have imagined. After all, in neither of his mayoral elections did Lindsay win an outright majority. Regardless of how progressive New York politics were in Lindsay’s day, the mood certainly shifted afterward. “Post-Lindsay, the fiscal crisis kind of did put a damper on the progressive left in New York,” Cannato says. “There was also a kind of political consensus after the fiscal crisis that there weren’t going to be big programs because we couldn’t afford them.”

Meanwhile, the city’s near-death experience led to broad acceptance of the notion that what’s good for Wall Street and the real estate sector—and only them—is good for the city. “The fiscal crisis reshaped politics in NYC,” says Baruch College professor Douglas Muzzio. “The constellation of players and their influence changed.”

Another critical factor has been race. Virtually every recent election in the city has been characterized by a pronounced racial polarization. At one pole is usually a Democrat supported by white progressives and most blacks and Latinos. His or her opponent gets the backing of most of the white population along with small portions of other racial blocs. Because progressives depend on a more heterogeneous ethnic mix, their coalition is harder to maintain, observes John Mollenkopf, a City University professor and leading demographer. Koch, Giuliani and Bloomberg all enjoyed solid support from working-class whites. Meanwhile, racial tensions between Green and Ferrer hurt progressives in 2001; blacks never rallied to Ferrer’s 2005 run; and Latinos weren’t galvanized by Thompson’s 2009 bid to become the city’s second black mayor.

The city’s population has changed over the past four decades, but the shifts haven’t necessarily favored the left. Whites may be a shrinking share of the city’s voting-age population, but they are still more likely than most other groups to vote. And the white population is continuing to evolve from working class to professional class, making the city’s whites more likely to be culturally liberal—i.e., supporting abortion rights and same-sex marriage—but less likely to feel strongly about economic issues like wages and inequality, around which a broader progressive agenda might be built.

Within the racial blocs, moreover, there are trends toward the right, according to veteran Democratic political strategist Hank Sheinkopf. The white population includes rising numbers of Orthodox Jews and Russians, often conservative. The number of African-Americans, traditionally liberal, is dwindling; the number of Caribbean-Americans, who are decidedly less liberal, is growing. “The demographics have changed,” Sheinkopf declares. “And demography is political destiny.”

In the background is a steady decline in voter participation. Historian Fred Siegel notes that turnout in mayoral elections fell by nearly 50 percent between 1993 and 2009. Dinkins benefited in 1989 from the energizing effect on black voters of Jesse Jackson’s 1988 presidential run. But Barack Obama’s White House win in 2008 did little for Thompson’s mayoral bid the following year.

* * *

The Politics of Fear

The bad news is that none of these trends offer much hope to progressives. The good news is, they don’t explain everything. After all, Dinkins did win in 1989. And before the 2001 terrorist attacks, the city was on the verge of electing either Ferrer—whose campaign revolved around a critique of inequality—or Green, a lifelong progressive.

Ferrer, now a consultant and member of the MTA board, says both the ‘89 and ‘01 races came at moments when the stars aligned to favor progressive change. “The city [in 1989] was on the brink of a mass nervous breakdown because Koch would play these divide-and-conquer games—crazy things. It just became unbearable,” he says. In 2001, “it was the Koch animus multiplied by the Giuliani animus. People for a while were saying, ‘We’ve got to be fairer to people. We’ve got to give people hope.’”

Incumbent mayors have a built-in advantage for re-election. Giuliani’s re-election bid in 1997, for instance, managed to neutralize the Democratic establishment and the media. “There were people of prominence who supported me but did not endorse me because they thought their standing in the city would be severely damaged,” says Ruth Messinger, Giuliani’s opponent in that race. “And the issues were not as well covered as I thought they would be because the press thought they knew the outcome.”

Putting those incumbency races aside, then, the pivotal losses for progressives were in ‘77, ‘93 and ‘01. In the first, the city was reeling from fiscal crisis, terrible riots and the Son of Sam spree. In 1993, there was a media frenzy about the murder rate. In 2001, New York was jolted by the worst terrorist attack in history. Fear links all three, Siegel notes: “There’s a sense of the city breaking down and people facing personal peril.” He adds, “Conservatives win when traditional liberalism breaks down. It only lasts as long as the crisis. Fundamentally, the people who are mobilized for politics in the city are overwhelmingly liberal.”

The lack of an apparent crisis in 2013 creates an opening for progressives. So does the return of conventional political warfare after three cycles of Bloomberg’s thermonuclear campaign spending. New York is an expensive city to run a campaign in, but a regime of spending caps and matching funds usually creates a level playing field. Bloomberg didn’t merely circumvent that system; he steamrolled it, spending $266 million to the $38 million spent by his opponents. Bloomberg’s “not a conservative. He’s not a liberal,” says Green. “He’s a bank.”

The dollar figures fail to capture the distortion wrought by the mayor’s money. Even as he donated millions to Republican organizations, directed billions in subsidies to developers, and rallied a Who’s Who of elites to push his bid for a third term, the mayor continued to scapegoat “special interests.” More than a few progressives bought it, assuming that the term referred to the NRA and tobacco companies, not noticing that the mayor also meant union members and parents with qualms about standardized testing.

As the mayor and his money exit, so does the mirage of New York run via noblesse oblige. This gives progressives a chance to present an alternative vision for governing.

* * *

2013: Could This Be the Year?

Some are skeptical that the left can do it. Sheinkopf believes progressives in the city have defeated themselves by failing to find a message that resonates with working-class people across racial lines. “The battles that progressives sought to win were causes that blue-collar people didn’t care about,” he says.

But the moneyed class seems to take the possibility of a progressive breakthrough very seriously. Amid palpable fear in the business community about life après Bloomberg, Wall Street has raced to the barricades. In December, the Rockefeller Foundation awarded the Brooklyn Chamber of Commerce a grant to, as The New York Times put it, “come up with ways to compel the candidates in next year’s race for mayor to pay more attention to business issues.” Around the same time, the city’s tabloids—owned by some of Bloomberg’s richest allies—began to hype a mayoral bid by MTA chair Joe Lhota, a former Giuliani deputy mayor. Lhota faces a billionaire supermarket magnate and two other opponents in the Republican primary, but the smart money says he gets the GOP nomination in a walk.

On the Democratic side, two contests are playing out. The September vote will select the nominee. But the campaign between now and then will determine if that nominee is prepared to challenge the unequal distribution of wealth and power that has prevailed in the city since the fiscal crisis. The Democratic candidates—Thompson, Quinn, John Liu, Bill de Blasio and Sal Albanese—have progressive elements in their DNA. But all face a question: How far are they willing to go?

Albanese, a long-shot candidate, wrote the city’s first living-wage bill as a councilman in the 1990s and, despite hailing from conservative Bay Ridge, supported gay rights. But so far, he doesn’t seem seized by a need for fundamental change in the city; he told me in January that “the city’s not in bad shape vis-à-vis other places.”

Thompson, the former city comptroller, benefited in 2009 from anger at Bloomberg’s term-limits reversal. He’s a sharper candidate this time but has done little to distinguish himself from the other hopefuls. Liu, the current comptroller, has no differentiation problem, having staked out the boldest position on wages (he wants a city minimum wage of $11.50) and stop-and-frisk (he proposes ending it, not mending it). But two former Liu associates are scheduled to go on trial soon over a campaign finance scandal in which, though Liu has not been charged, his reputation has been damaged.

Public Advocate de Blasio has articulated the most cogent critique so far of the Bloomberg era, recently chastising the mayor for having “transferred more public value to the private sector in the past twelve years than at any time before in our history.” While he has avoided some key battles—notably, he defended the NYPD’s broad spying on Muslim communities—he’s been out front on others, including the fight to preserve the earlier term limits in 2008. Right now, he is running a distant second to Quinn in campaign fundraising and far behind her (with the rest of the contenders) in the polls.

Quinn, a onetime housing advocate, would be the city’s first woman mayor and the first openly gay mayor. On many issues, she is reliably progressive: battling against anti-abortion organizations that bill themselves as “crisis pregnancy centers,” defending the so-called “Ground Zero mosque,” trying to stop the city from fingerprinting food-stamp applicants. Depictions of her as the mayor’s lapdog overstate the truth: Quinn has broken with the mayor many times, overriding at least twenty-seven mayoral vetoes.

On several big-ticket issues, however, Quinn has protected the mayor’s and the business sector’s interests: watering down to the point of comedy a living-wage measure and facilitating Bloomberg’s odious term-limits overhaul. Her affordable housing plan revolves around a massive tax break for developers, and she says she wants to reappoint Police Commissioner Ray Kelly, whose civil liberties record is poor.

Very recently, Quinn made two high-profile moves away from the mayor. She backed an inspector general for the NYPD and, after months of pressure from other candidates, agreed to allow a vote on a weakened but still valuable bill to require paid sick leave for employees of most city businesses. These shifts suggest that Quinn is still figuring out how far left she needs to move to win the primary without losing the business support she’ll need in the general. Where that needle lands could be where New York progressives’ prospects stand for the next eight years.

In recent surveys, Quinn polls close to 40 percent, the share of the primary vote she needs to avoid a runoff. It’s early, though—a quarter of the voters haven’t picked a candidate, and it’s possible another will enter the race (former Representative Anthony Weiner, as this article went to press, let drop that he is mulling a run). But if the 2013 campaign does come down to Lhota versus Quinn, the smart and brassy speaker will be progressives’ best hope.

But not their last. While New York City has a “strong mayor” system, the City Council has significant ability to check the executive’s power. The council now boasts a Progressive Caucus, with one member, Melissa Mark-Viverito, a candidate to succeed Quinn as speaker and another, Brad Lander, earning national renown for progressive advocacy. Whoever is elected mayor, the council will likely be a more effective progressive counterweight.

Meanwhile, progressive forces outside government are gearing up to fight for their vision of the city. Organizations like New York Communities for Change, the Community Service Society and the Center for Popular Democracy have been organizing mayoral forums and issuing policy reports aimed at forcing issues of economic and social fairness onto the campaign radar screen.

Winning City Hall or the policy battles of 2014 and beyond won’t be easy. But Messinger, who now heads the American Jewish World Service, is hopeful. “It’s always impossible to elect a progressive… and then you do it. It depends on people not feeling overwhelmed or depressed, but trying to figure out what strategies they can bring to bear to mobilize the electorate.”

That’s where my friend, the progressive operative, was wrong in 2009. A fight doesn’t have to be fair for you to win it.

Writing in 2009 in the eve of Michael Bloomberg’s third term, Richard Kim decried progressives who might vote for the billionaire mayor, saying, “Poor, hungry New Yorkers will be stripped of food stamps that the federal government says is both necessary and good stimulus, while the bollocks-for-brains bankers who got us into this mess will get office space and taxpayer moolah to restart the cycle of speculation.” Read all of the articles in The Nation‘s special issue on New York City.