If Democrats want to make the case that Dr. Ben Carson is unqualified to be secretary of the Department of Housing and Urban Development, they can use the words of Carson himself: “Dr. Carson feels he has no government experience; he’s never run a federal agency,” his friend Armstrong Williams told reporters, when rumors of Donald Trump’s plan to put Carson at HUD first emerged. Carson told Trump, “I preferred to work outside of government as an adviser.” But on Monday, Trump tapped Carson to head the $47 billion agency that oversees home-mortgage lending, public-housing administration, desegregation efforts, and fighting housing discrimination.

Late in his campaign Trump frequently took to asking African Americans living in the “inner city,” “What the hell do you have to lose?” They may be about to find out.

Carson’s selection is easy to mock, especially given the way Trump mocked him during their primary campaign. The eventual GOP nominee ridiculed Carson for his warnings of his own “pathological temper,” comparing him to a child molester. “I don’t want to say what I said, but I’ll tell you anyway,” Trump told his supporters. “I said that if you’re a child molester—a sick puppy—you’re a child molester, there’s no cure for that.” He reenacted Carson’s story of trying to stab a friend only to have his belt buckle block the blow. “Give me a break, the knife broke?” Trump declared, twisting his own belt buckle. It was a cringeworthy display of Trump bullying. When Carson dropped out of the race and endorsed Trump, all was forgiven, on both sides.

Democrats are signaling a fight over Carson, however. Democratic House minority leader Nancy Pelosi called him “a disturbingly unqualified choice.” Oregon Senator Jeff Merkley agreed, saying he was “deeply concerned that the incoming Administration has proposed someone who believes ‘poverty is really more of a choice than anything else’ to lead American urban development. Telling people that their lack of means is their own problem is not a solution.”

One of the things that makes the choice of Carson painful to fair housing advocates is that hidebound HUD has made “fundamental shifts” during the Obama years, in the words of PolicyLink’s Angela Blackwell. (Full disclosure: I’m on PolicyLink’s board.) “The big shift that took place is in understanding that housing has become a proxy for opportunity.” HUD’s mission, therefore, became broader, and the agency began to collaborate with others targeting economic opportunity, to think about access to transportation, jobs, grocery stores, schools.

Though he grew up poor in Detroit, it’s unlikely Carson knows much about the evolution of HUD or other federal agencies. The 65-year-old surgeon has had little to say about urban policy during his career. But last year, as he ran for the Republican presidential nomination, he published an op-ed that criticized some of HUD’s more innovative desegregation efforts, arguing that “government-engineered attempts to legislate racial equality create consequences that often make matters worse.” Also during the campaign, he attacked a program to relocate Chicago residents to middle-class areas in Dubuque, Iowa. “This is what you see in communist countries, where they have so many regulations encircling every aspect of your life that if you don’t agree with them, all they have to do is pull the noose. And this is what we’ve got now.”

Carson attacks HUD programs as “social engineering,” like your typical conservative. He ignores that it was very deliberate social engineering that created the patterns of housing segregation and inadequate opportunities that have disadvantaged working-class and poor people of color over generations. Restrictive covenants that excluded non-whites; redlining poor areas so they couldn’t receive mortgages or insure property; highway policies that obliterated city neighborhoods and spread housing opportunity to the distant suburbs, unreachable without a car (and often inhospitable to those who weren’t white); this is what generations of discriminatory “social engineering” created, which HUD has been trying to undo for the last 50 years.

Ironically, in 1973 HUD joined up with the Justice Department to sue Trump and his father for discriminating against African Americans in their rental units. It’s hard to imagine the agency mounting any such challenge under Carson. Urban historian Tom Sugrue compares Carson’s likely reign at HUD to Samuel Pierce’s under Reagan: Both men were the Republican presidents’ token black appointments, heading a department that’s largely viewed as a “black” agency (you see “urban” right in the title). Pierce oversaw the gutting of federal support for low-income housing subsidies, which fell from $26 billion to $8 billion under Reagan. He also worked to privatize agency holdings, which became a festival of crony capitalism, Sugrue writes. The omnipresent self-dealer Paul Manafort lobbied for $43 million subsidy for a rundown New Jersey project, pocketing a fee of $326,000. Eventually 16 HUD staffers and consultants were convicted of fraud for their work; Manafort was never charged. Imagine inexperienced Ben Carson presiding over another spree of Trump-inspired development cronyism using the agency’s billions.

“The two biggest risks I see with a Carson administration are 1) that HUD sits on the sidelines, allowing Congress to chop funds for affordable housing, while our nation’s housing problems get worse,” said Sarah Edelman, director of housing policy for the Center for American Progress, via e-mail. “And 2) HUD abdicates responsibility for enforcing the Fair Housing Act, which would leave women, families with children, people with disabilities, and people of color less defended against a landlord or mortgage lender who discriminates against them and stall important economic mobility work.”

Blackwell shares those concerns, and others, but holds out hope that, since Carson “seems to be a person who’s concerned about outcomes,” he might be helped to see that HUD “can be the opportunity agency.” I admire her modicum of optimism, but I haven’t found a way to share it yet. Still, it’s possible the career employees of HUD who’ve internalized a new way of looking at housing will be unable to unlearn it, and that some of the Obama-era culture will remain. But, as Sugrue observes, “The possibilities for corruption are legion with a HUD secretary who knows nothing about the agency, its programs and its vastly complex budget.” Especially with a callous real-estate developer as president.

Let’s hope Democrats don’t let Carson slip by, given that worse nominees—Jeff Sessions for attorney general, Tom Price for Health and Human Services, and Steve Mnuchin for Treasury secretary—must be fought hard. His confirmation hearing must get to the heart of HUD as a potential “opportunity agency”—and discover whether Carson is interested in offering that opportunity to low-income Americans served by its programs.