If the cascade of #MeToo posts across social media has clarified one thing, it’s that sexual-assault and -harassment victims have no profile. Those speaking out have been young and old, representing all gender expressions, those dressed modestly and those whose clothes are more risqué. Sometimes victims are as conventionally stunning as the actresses who stepped forward recently to talk about Harvey Weinstein—but more often, they’re not. Nothing unites survivors except their vulnerability to their assailants.

But what we have also started to see is that too many perpetrators do have something in common: They’ve been given power, and they have a tendency to abuse it.

In particular, this year we’ve seen dozens of allegations against people with a particular kind of power: Harvey Weinstein and Donald Trump, and Roger Ailes, and Bill O’Reilly, and Bill Cosby and other, less world-famous people who were demigods in smaller music, media, activist, and tech scenes. These people have all reached a position where they hold other people’s careers and reputations in the palm of their hands, and they are surrounded by enablers who worship their status. We go to the movies to watch Meryl Streep’s fashion-editor and Jeremy Piven’s movie-agent characters scream at and harass their assistants—and think it’s funny. Yet we now know that plenty of real-life fashion moguls and Hollywood big shots appeared to have enabled Weinstein.

To push back against a culture that fosters harassment and abuse, we ought to be suspicious of such lopsided power hierarchies, on larger scales and also on smaller interpersonal ones—like a local clergyperson who is deferred to by the congregants, the charismatic acquaintance who gets pushy when drunk and doesn’t catch flak, or a co-worker who makes racist comments with impunity.

In many of these cases, it’s crucial to use our voices if we safely can, whether loudly or in whisper networks to warn others of bad behavior: This tips the power balance, just a little bit. But real change starts further up the chain. It’s time to think about the qualities that are valorized in our own communities, living spaces, and workplaces, and to push back on the ingrained idea that domination and aggression are “winning” traits worthy of promotion and support. We have to grapple with the fact that the qualities that accompany harassment and abuse often overlap with qualities that are widely admired—or at least tolerated—in the name of success.

The story of Harvey Weinstein has a lot to teach us here. Last week, a group of his employees released a joint statement noting that industry norms allowed their boss to behave awfully: “We treat these abusive people and places as rites of passage, instead of with the disgust they deserve,” they wrote. “Harvey Weinstein is far from the only sociopathic bully we’ve exalted over the years. Employees who work under our industry’s most notorious bosses are regularly asked to surrender their dignity in exchange for professional success.”

Weinstein in particular was someone who, as screenwriter Scott Rosenberg noted, seemingly owned an entire industry, including the very people who were supposed to advocate for those he hurt: “And if [his victims] discussed it with their representatives? Agents and managers, who themselves feared The Wrath Of The Big Man?” Rosenberg wrote on Facebook. “The agents and managers would tell them to keep it to themselves. Because who knew the repercussions?”

Weinstein, a titan in his world, may have kept his most illegal abuses (somewhat) veiled—but he was widely known to be terrifying to deal with. Actress Kate Winslet recently came forward to talk about the trials of working with him on the film The Reader. “He was bullying and nasty,” she said. “He used to call my female agent a [vulgar name for a woman] every time he spoke to her on the telephone.” Winslet describes a moviemaking process that was subject to his petty whims and ego. In her account, it was clear Weinstein could throw a switch and end a project that was important to dozens of people. And he took advantage of it.

What Winslet, Rosenberg, and the Weinstein employees are talking about is Weinstein’s accepted public behavior that, while not illegal on the surface, was not professionally appropriate—in fact, to use Winslet’s words, it was downright bullying. Yet, as Rosenberg acknowledged, this was given a pass, even rewarded, because the man made money and won awards. He made things happen for people. He was a mover and a shaker. He was admired.

He had power without accountability.

Who else does that sound like? Consider the night that presidential candidate Donald Trump hosted Saturday Night Live. The public was not yet aware of the now-notorious Access Hollywood “pussy” tape, in which he describes groping and assaulting women. But the man had just made a series of comments about Mexicans’ being rapists and murderers that was so offensive and racist that it had prompted picket lines outside the NBC studios.

It was far from the first deliberately cruel statement he had made: He had run ads against the falsely accused Central Park Five demanding the death penalty, made crass and disturbing statements about women on Howard Stern’s radio show, and had a demonstrated history as a predatory and racist landlord. All this showed a predilection for taking advantage of people with less power. And now Trump was bringing that spirit to the campaign trail: He was leveraging his spotlight against vulnerable minority groups and against individuals who criticized him, too.

But none of this bullying was met with real consequences, or shunning. Sure enough, the show went on—SNL gave him airtime for free, Jimmy Fallon later tousled his hair. And we all know the rest: Time and time again Trump flexed his ability to hurt other people, and was allowed to, even praised by some. So, naturally, when he was eventually accused of assault and harassment by dozens of women, it made little difference. He now sits in the most powerful seat in the world, insulated from having to be responsible for his actions.

Bill O’Reilly is a third example. He used his nightly news show to viciously lay into people he didn’t like, including repeatedly going after Dr. George Tiller, an abortion provider who was eventually murdered by a fanatic. Even Bill Cosby, who projected “niceness” on the surface (as many bullies do at first), was accused by Janis Ian of essentially trying to blacklist her because of an assumption he made about her sexuality.

We need to make it socially unacceptable to treat other human beings this way, and we need to be wary of ceding this much power to single individuals. Those in positions of hiring, promoting, funding, and otherwise shaping their communities need to take responsibility for the values they tolerate and reward.

No, being a threatening jerk is not illegal, and, yes, sometimes behaving like a bully can “get things done.” We should not conflate being a jerk with abusive or criminal behavior. But at the same time we should recognize that the free pass given to figures like Trump and O’Reilly in public is a cornerstone of a culture that allows harassment in private. So by making casual abuses of power less acceptable, we can push back on the culture of tolerance overall—because where bullying and domineering behavior is widely tolerated, abuse is likelier to go unchecked.

I can’t help thinking about some of my own tangential experiences with harassment: the way one particular boss, for instance, chose to sexually target some of his employees and harangue, bother and undermine others. He attempted to exert dominance over all of us, in other words—just in different ways. It’s all too common: A recent survey showed that “two in five LGBT workers (40 percent) report feeling bullied at work, 11 percentage points higher than the national average of all workers combined.” One can imagine that in workplaces where bullying in meetings is allowed, people get the message: You can mess with or even harass your colleagues without repercussion.

A persistent media narrative posits that sexual assault and harassment is linked to sexual desire, or something inherent in masculinity—the president used the phrase “locker-room talk” to dodge the issues brought to light by his tape. If it’s about sex, of course, people are tempted to respond by policing sex: Pundits take aim at casual office gatherings and women’s clothing choices. But these stories of real harassers show us it’s not really about sex. And it’s not entirely about masculinity, either, as recent allegations against a prominent female entrepreneur, the founder of THINX, show. Demeaning behavior is committed by both genders when the power imbalance is there.

We equate doing something well as being “like a boss.” But what are the characteristics that make up a “boss”? Our culture thrives on exalting people with “alpha” personalities: entrepreneurs, people who push boundaries, “winners” and those who “don’t give up”—we like people like Harvey Weinstein because power, and its perks, attracts us: “Harvey was showing us the best of times,” wrote Greenberg. “He was making our movies. Throwing the biggest parties.”

Of course, the only way to stop abusive behavior is for abusers to refrain from hurting people—and it should be noted that plenty of predators may seem sweet and innocuous at first rather than being belligerent like the men above. We can’t and shouldn’t put the onus only on ourselves, and we should never expect people who are vulnerable and intimidated to carry this burden.

But one small thing those of us with a little leverage can do to shift the culture and be better allies is limit the praise and appreciation we offer to people who exploit power dynamics in obvious, less “creepy” ways. This applies most specifically to men and other people with privilege, who can fit more easily into cultures where dominance is valorized. The truth is, if you can take part in such a culture easily, it’s a sign that you should resist it. And one way to do so is condition ourselves and our communities to think about sexual assault and harassment as linked to bullying, domination and cruelty—rather than as cases of “sex gone wrong.”

We can support changes like strong unions that prioritize harassment as an issue, and give workers a space to openly discuss problems they face in the workplace. We can push for, or conduct ourselves, anonymous surveys about bullying and company culture combined with transparency about harassment numbers in the workplace. We can fight for a social safety net that allows people to leave their jobs without worrying about destitution, and leadership structures that utilize consensus and democracy over autocratic decision-making. We can show more solidarity with the people most likely to be harassed because of the power their bosses have over them: low-wage workers and undocumented immigrants. And we can shift our mindsets to think positively about behavior we code as “feminine”—empathy, thoughtfulness, listening—because those, too, are qualities that belong to real leaders, and winners.