Greg Billings stood in the cold outside the National Republican Club on Tuesday, holding a photo of current House majority whip Steve Scalise and, just behind his left shoulder, Ku Klux Klan wizard David Duke. “Is this your GOP?” Billings’s sign read. Somewhere inside the club Scalise was due to meet with Republican donors.

“Those type of people having access to power is disturbing to me. I don’t like people with those connections making policy decisions,” said Billings.

News that the third-ranking Republican in the House addressed a group of white supremacists in 2002 broke in late December; since then we’ve learned that the Louisiana representative also voted multiple times against declaring Martin Luther King Jr.’s birthday a national holiday, and tried to squash a resolution apologizing for Louisiana’s role in slavery. Regarding the first revelation, Scalise claims not to have known he was addressing a room full of racists. Civil-rights advocates, lawmakers and even several conservative commentators have called for him to step down or at least demonstrate some commitment to racial equality. So far, other GOP leaders are standing by him.

How shocked, and how outraged, should we be by the “Scalise episode,” as Washington Post columnist Dana Milbank called it? Milbank offers little of either emotion, claiming that it is really “an occasion for some pride, and an indication of how much progress has been made in marginalizing the hateful, even in conservative politics and even in the Deep South.” Milbank sees the Scalise episode as evidence of a “transformation,” which seems to mainly hinge around the fact that these days fewer lawmakers are giving speeches to white supremacist groups.

That’s all well and good; it’s also an absurdly low bar to which to hold our elected representatives. If it’s hard to muster shock and outrage towards Scalise, perhaps it’s because the decision to keep him in a leadership position reflects values the GOP routinely asserts in its policies. Is it really surprising that a figurehead of the party that actively seeks to deny health insurance to a disproportionate number of black Americans, deport as many immigrants as possible, and strip away the right to vote once rubbed shoulders with white supremacists?

“It’s not going to stop,” Greg Billings said outside the Republican National Club when I asked if removing Scalise would fix the GOP’s race problem. “It’s policy as well.”

It’s true that the GOP is putting a lid on people whose rhetoric might ruffle feathers. The latest person to muck up the rebranding effort is Dave Agema, a member of the Republican National Committee. Agema recently posted an essay from a white-supremacist newsletter to his Facebook page which argued that “blacks are different by almost any measure to all other people. They cannot reason as well. They cannot communicate as well. They cannot control their impulses as well. They are a threat to all who cross their paths, black and non-black alike.” Agema deemed the article “very enlightening.”

RNC chairman Reince Preibus called on Agema to resign, saying, “The tone and rhetoric from Agema is consistently offensive and has no place in politics or any rational conversation.” For several years now Preibus has been leading calls for the GOP to broaden its tent, to soften its language, and to invest in black communities in order to court voters there. Yet he’s still holding a torch in the voter-fraud witch hunt—in essence, saying he wants to convince people of color to vote for conservatives, while supporting the most aggressive assault on their right to do so since the Jim Crow era.

In this context it’s hard to accept the GOP’s “talk pretty” project as real progress. Voter suppression is just one of the policies pushed by mainstream Republicans that are “consistently offensive,” as well as tangibly damaging, to Americans of color. These policies impact the quality of peoples’ lives today more than Scalise’s meetings or Agema’s Facebook posts, but few raise the same stench of scandal. Keeping Scalise in a leadership role “says a lot about what the conference’s priorities and values are,” White House press secretary Josh Earnest noted. Yes, but one can interpret those priorities and values regardless of where he sits.

Here’s a recent example. In the waning days of 2014 congressional Republicans attached a rider onto a must-pass budget bill intended to bar Washington, DC, from setting up a regulated marijuana market. Voters in the District overwhelmingly approved a legalization measure in November, after a campaign that was striking in its focus on racial justice. Nine of ten people arrested for drug offenses in the District between 2009 and 2011 were black, though black Americans are no more likely to use drugs than whites, and make up only half of the DC population. By overriding the will of DC voters and the city’s government, Republicans went out of their way to maintain a transparently racist drug enforcement system.

It’s also worth noting that even as Republican leaders try to tone down their colleagues’ language, some people are still fair targets for discriminatory rhetoric as well as policymaking. It’s possible to talk about children of immigrants with “calves the size of cantaloupes because they’re hauling 75 pounds of marijuana across the desert,” for example, and be taken seriously in related policy meetings

So yes, the GOP has a race problem—but it’s a policy problem, not just a PR problem, and people like Scalise and Agema are a visible but very small part of it. Removing Scalise from leadership is the very, very least the party could do to signal that it takes the concerns of nonwhite voters seriously. “David Duke without the baggage”—Scalise’s one-time slogan—is a pretty good motto for what the modern Republican Party seems content to become. Get rid of the bags, but keep what’s inside.