A few months before I met Vivian Warner, she got the call she had been waiting so long for that she’d forgotten to hope for it. It was Baltimore Housing, the agency that oversees subsidized housing in the city. After four years on the wait list, Vivian would receive a housing voucher, and could finally move off of her sister’s couch into her own home. A few weeks later, Vivian boarded a bus with the other lucky winners and drove around the city to visit eligible homes. At the last stop, the bus pulled up in front of a low-rise apartment complex. It was not quite what Vivian had imagined, but there was a two-bedroom available, and Vivian would pay just $55 a month out of pocket from her part-time income. She signed the lease that afternoon.

The housing voucher Vivian waited years to receive is part of the federal government’s most recent attempt to house the poor. Since the 1930s, it has employed housing assistance as a key tool in its war on urban blight and poverty. But these attempts have often failed those who they were meant to protect, at times recreating the very inequality they intended to undo.

Postwar, the Federal Housing Administration underwrote home loans for millions of white Americans, while banks systematically denied them to black families, a process called redlining. Even in federally funded public housing, the poor had no respite from the marginalizing forces of inequality. In the 1950s and ’60s, high-rise public housing was erected in neighborhoods that already suffered from segregation, underinvestment, and decline. And when the Fair Housing Act of 1968 outlawed housing discrimination by “race or national origin,” local housing authorities in cities like Chicago, Baltimore, and Dallas continued to keep two separate housing lists: one for whites, and one for blacks.

Vivian is part of a generation of poor urban dwellers who left the concentrated poverty of high-rise public housing towers, which by the 1990s were crumbling from neglect. Across the country, the buildings were torn down, and along with them an entire system for housing the poor was dismantled. In the place of public housing, the federal government needed a new solution, one that would remedy the concentrated poverty and segregation it had helped to create.

This solution expanded an existing program relying on the private market to house the poor: housing vouchers. These vouchers make up the difference between what a needy household can afford and the cost of a unit in the private market. They are meant to allow families to rent in any affordable neighborhood, offering men and women like Vivian their very first chance to choose where to live. Today, of the 5 million households across the country receiving some form of federal housing assistance, over half now live in privately owned properties, many through the Housing Choice Voucher program, previously called Section 8.

The program has undeniably positive effects. Vivian was able to regain custody of her twin boys thanks to her new home. For Tony Young, a 55-year-old man with HIV, receiving a voucher meant relief from the cold, hard bed under the bridge where he slept when there was no room at the homeless shelter. Joann Jones, a young mother of two, was able to buy fresh fruits and vegetables for her 7-year-old at the local store while he attended a high-performing public school nearby. The basic economic relief that vouchers provide cannot be understated. And they also give families something more: flexibility in times of crisis to respond to the demands of their jobs, their children’s needs, even the whims of landlords. By letting recipients choose where to live, vouchers confer dignity and affirm a sense of belonging potentially free from the stigma of “public” housing. Most important, they may help people to realize their dreams of a place to call home.

By untethering federal housing aid from the disadvantaged neighborhoods to which it was once attached, vouchers offer millions of poor Americans the opportunity to move to a new neighborhood where streets are safe, schools have resources to teach their children, and jobs are bountiful. But not everyone does. Vivian, for example, might have used her voucher in a number of safer, more affluent neighborhoods. But time and resources to find an ideal home are limited. And America’s long history of discriminatory housing practices have shaped the residential landscape in ways that cannot be undone by simply offering families a “choice.” Though voucher holders have moved to areas that are less poor than the ones available in the heyday of public housing, many are reconcentrating in poor neighborhoods.

This reconcentration matters for a reason that social scientists like William Julius Wilson have long known to be a fact, but they finally have the hard numbers to prove: Where you live matters. It matters for your quality of life, for how much money you make in your lifetime, and for your children. Raj Chetty’s new work shows that a child growing up in a city like Baltimore will make 14 percent less over his lifetime than one in a typical American county, even after accounting for individual factors like income and education.

If where you live is so crucial, then we ought to pay attention to the role housing policy plays in where families end up. In a landmark case this past July, the Supreme Court ruled that housing discrimination need not be intentional to have harmful effects of segregation. This is the first time the legal concept of “disparate impact”—the idea that a policy may disproportionately affect certain groups even absent injurious intent—was applied to federal housing policy. The decision substantiates an important change in the way discrimination persists in contemporary America: We are moving away from the overt racism of Jim Crow, toward one maintained by enduring institutions that inadvertently perpetuate longstanding inequalities—a “racism without racists.” This shift is crucial to understanding how and why racial inequality continues to plague our nation.

Housing vouchers offer a chance to remedy this disparity, but are not yet equipped to fully do so. Not all voucher holders succeed in finding a place to live, and those who do are often unable to find homes in neighborhoods that have jobs and good schools. And although vouchers are a potential tool to dismantle concentrated poverty and segregation, it turns outs that black voucher holders live in neighborhoods that are poorer and far more segregated than those of white voucher holders, revealing the program’s shockingly disparate impact on white and black families. If black voucher holders face obstacles that prevent them from using their vouchers in the same neighborhoods as whites, then something needs to be done.

In their current form, vouchers fail to take account of an important lesson: A roof is not enough. Where you live matters. Vouchers shouldn’t merely keep people off the streets; they should help families move to neighborhoods with more opportunities. What can we do then to make the voucher program work better to reduce inequality? There are a number of policy fixes to reduce barriers that prevent families from using their voucher in low-poverty, integrated neighborhoods. For example, we could do a better job providing mobility counseling and transportation to help families explore new neighborhoods. There are also solutions related to landlords, like passing national legislation that makes it illegal to discriminate against someone who pays their rent with a voucher, and other policies that would encourage landlords in low-poverty areas to accept housing vouchers.

Even with these fixes to modify the disparate impact vouchers often have, it is not enough to simply move the poor out of poverty-stricken neighborhoods. It is imperative that we address the root causes of poverty and inequality by implementing change at the level of the neighborhood itself, improving the environments around poor families by investing in schools, institutions, and the economy. But this systemic transformation cannot take place overnight, and it will face stark political opposition. It remains to be seen how the political climate of the next presidency will unfold to potentially make good on the Fair Housing Act’s recently renewed half-a-century-old promise to “affirmatively further fair housing.”

While we wait for political change, we can act to undo the disparate impact this program has on minority families, who don’t fully reap its rewards. Housing vouchers could be a powerful instrument to remedy the indelible dangers of living in a poor environment, for families of all backgrounds.