There’s been a flurry of online conversation around domestic issues lately—be it the politics of staying-at-home or how we’re still trying to “have it all” to no avail. Things have gotten heated.

There’s much missing in the framing of these debates—from the expectation of power and privilege to a limited idea of what success is. What’s irked me is the continued assumption that this is a women’s issue. The problem isn’t that women are trying to do too much, it’s that men aren’t doing nearly enough.

A new report from the Bureau of Labor Statistics shows that women—even those with full-time jobs—still do the bulk of housework and childcare. On an average day, 48 percent of women and 19 percent of men did housework. Married women with children who work full time spend 51 minutes a day on housework while married men with children spend just 14 minutes a day.

The breakdown of childcare responsibilities was not much different—55 percent of working men said they cared for their kids on an average day, whereas 72 percent of working women did. Women also reported spending more time during the day caring for their children than men.

This isn’t news to most; statistics (and feminists!) have long showed that women work a second shift at home. But despite the glaring inequality on our doorstep—and in our kitchens—the recent debates do little to address tangible ways men can be held accountable. Sure, they’re mentioned as an aside every once in a while—It’s good to have a supportive partner! When men “help out,” life is easier!—but men’s participation in the domestic sphere is largely discussed as optional, while women’s is assumed to be mandatory.

I’ve seen straight, partnered women explain their decision to stay-at-home by noting that childcare would have taken too much out of their paycheck—as if this cost was just theirs to bear! Or couples who call a woman’s decision to quit her job a “personal” issue, while in the same breath noting that it was because her salary was lower than her husband’s. (The last time I checked, the wage gap was a political issue.)

But even more dangerous than the “I choose my choice” brush-off that tends to surface when someone takes the politics of housewifery to task, is the contention that women want to be doing all this work. That we are naturally inclined towards things domestic—especially caring for our children. Perhaps for some women this is true; but the generalization hurts all of us. After all, how can we effectively fight for workplace policies if the presumption is that when push comes to shove, we don’t really want to be there?

Anne-Marie Slaughter, for example, pays lip service to the importance of men’s participation in parenting (again, as if it’s optional) but comes to the conclusion that it’s just different for women. In discussing the oh-so-radical notion of equality and co-parenting, Slaughter writes that women are just not as comfortable as men being away from their children.

I’ve come to believe that men and women respond quite differently when problems at home force them to recognize that their absence is hurting a child, or at least that their presence would likely help.… Many factors determine this choice, of course. Men are still socialized to believe that their primary family obligation is to be the breadwinner; women, to believe that their primary family obligation is to be the caregiver. But it may be more than that. When I described the choice between my children and my job to Senator Jeanne Shaheen, she said exactly what I felt: “There’s really no choice.” She wasn’t referring to social expectations, but to a maternal imperative felt so deeply that the “choice” is reflexive.

Dismissing socialization and gender roles as piddling compared to this amorphous idea of “maternal imperative” is part of the reason progress is stalled for family-friendly policies. I don’t believe we must ignore how much we love our kids and want to be with them in order to effectively fight for better parenting policies—but the assumption that women want to be mothers above all other callings in their life directly impacts the way we talk and work on these issues.

If we accept the gendered narrative that says women do care work and stay-at-home because it’s fulfilling—rather than because it’s necessary—then we support the idea that it’s women, not men, who should be doing the bulk of domestic work and that we need no policies to support us. Because, hey, taking care of our kids is reward enough!

This is why when the Census Bureau puts together its annual report on childcare, men’s caretaking is counted as “babysitting”—only mom’s care work is considered parenting.

When these conversations only focus on women—when men are mentioned as an aside, rather than a central part of creating change—we not only do a disservice to the American men who want more work/life balance but let those who benefit from unpaid female labor entirely off the hook from doing their fair share.

This isn’t about wanting “it all,” it’s about wanting fairness and justice—something that’s only possible if we radically change the gendered expectations of parenting. Anything less will keep us talking in circles.