Some years ago, while we were getting ready to move out of state, my husband and I held a garage sale. We’d advertised it in our community newspaper and on flyers around the neighborhood and had a huge turnout as a result. Dozens of bargain hunters milled about, asking about this or that item. “Cuanto quiere usted por el sofá?” an older gentleman asked me, pointing to our old green couch. I quoted him a price, adding, “Es un sofá cama.” Hearing our exchange, a white woman turned around and yelled, “Speak English! You’re in America.” “Hey—” I said, but she walked away in a huff, got into her car, and drove off.

I’ve been thinking about that moment, and the fiery anger behind it, as I hear about incident after incident in which white people lash out at people of color in public spaces. There’s the white lawyer who berated the workers at a Manhattan deli for speaking Spanish—insisting that “I pay for their welfare. I pay for their ability to be here. The least they can do is speak English”—and then threatened to report them to Immigration and Customs Enforcement. There’s the white student who reported a black student for taking a nap in the common room of their dorm at Yale, saying, “I have every right to call the police—you cannot sleep in that room.” And there’s the white mother who called the police about two Native American students taking part in a campus tour at Colorado State University, telling the dispatcher that “they are not, definitely not, a part of the tour.”

The language in these complaints—“I pay,” “I have every right,” “they are definitely not”—is quite illuminating. It indicates a belief on the part of these white people that they are the custodians of public space and can enlist the police to enforce its boundaries. The offenses committed by people of color are arbitrary and nearly limitless: waiting too long at a Starbucks in Philadelphia, having a barbecue on Lake Merritt in Oakland, playing a leisurely game of golf at a club in Pennsylvania, checking out of an Airbnb in Rialto, California. And once police officers get there, anything can happen, ranging from an arrest on charges of trespassing to the installation of a police perimeter and the arrival of a police helicopter.

To be sure, the belief that public space belongs exclusively to white people is not new, and this redlining has been inflicting trauma on people of color for a long time now. Whether it’s on the street, in a café, or at an airport, the visibility of people of color in public is tolerated only so long as it does not disturb the comfort of the dominant group. But the ubiquitous presence of smartphones with cameras has helped to document such incidents, and social media have brought them to national attention. That’s a useful development: The assertion of private authority over public space now comes with a social cost.

That is what happened to Aaron Schlossberg, the Manhattan lawyer who threatened to call ICE on the Spanish-speaking food workers and customers in New York. His law practice soon plummeted in customer ratings; he was hounded by reporters seeking comment; and his corporate landlord terminated his business lease. Protesters even brought a mariachi band to perform outside his apartment building. The public shaming that followed his rant could have a salutary effect: Maybe, just maybe, racists will think twice before making frivolous reports or issuing threats.

Few people have come to Schlossberg’s defense, yet there are some who say that the popular outcry against him is “unnerving” and constitutes harassment by modern mobs who demand nothing less than “conformity of thought.” Online mobs are scary, no doubt about that. But Schlossberg’s rant doesn’t amount to a civilized difference of opinion; it’s racism, pure and simple, followed by threats.

Schlossberg’s assertion of authority over public space is, of course, protected from government interference by the First Amendment. But that right doesn’t protect him from the social consequences of his speech, including disruption and discomfort. Those protesting Schlossberg’s actions are, in fact, exercising their own free-speech rights to object to his racism and nativism. The simple truth is that if racist behavior is insulated from social shaming, it will likely continue and multiply until it becomes accepted. What happens when a majority of Americans hold views like Schlossberg’s?

The history of this country is replete with examples of how public space was regulated to ensure that one racial group was made comfortable at the expense of others. This is why it’s important to speak out, and speak out now. Allies can help to stop the harassment, or at least deflect it. In the cell-phone footage of Schlossberg’s rant, for example, an Asian man can be seen interposing himself between the lawyer and one of the Spanish-speaking women he’s verbally abusing. In the Philadelphia Starbucks incident, an older white man repeatedly challenged police officers about why they were arresting the two black men when they’d done nothing wrong.

At the garage sale that day, after the woman took off in a huff, I turned to my husband in disbelief. “Did you just hear what that lady said?” I asked him. This question, I now realize, was an attempt at documenting the moment by having a witness for it. It was my first intimation that people’s relationship to public space is political, and that some of us move through it under surveillance by others. “I heard,” my husband replied, and then told me of many similar experiences he’d had as a Cuban American here in Los Angeles.

But public space belongs to everyone. If racists don’t like hearing Spanish being spoken in a deli, or having Native American teens on a campus tour, or seeing black folks going on about their lives, they should just stay home.