The recession seemed to mark the dying gasp of blue-collar jobs: as construction and manufacturing tanked, it sped along the decline of solid factory work for middle-class American men. That intensification of a long-term trend inspired such hard hand-wringing that there was a crisis of raw knuckles. Less noticed, however, has been a similar hollowing out in the middle for pink-collar work. Beyond the decent-paying public sector jobs that women have been kicked out of thanks to budget cuts in the recovery period, there’s another reason women are losing middle-class jobs and taking new ones as clerks and waitresses: job losses among secretaries and administrative assistants.

Back in 2011, a question perplexed many, even the likes of the Pew Research Center: why were women losing jobs in the private sector? Teachers were being fired because state budgets were so crunched, but that couldn’t explain why private sector women were falling behind men. My Roosevelt Institute colleague Mike Konczal and I took a look at the job losses by occupation and found a stunning trend: women had lost a total of 925,000 jobs in the Bureau of Labor Statistics’s “office and administrative support occupations” category.

Checking in a year later, a new report from the National Employment Law Project shows this trend is continuing. It finds that mid-wage jobs—those making $13.84 an hour to $21.13 an hour—have been decimated during the recovery, while job gains have happened mostly in low-wage jobs that make between $7.69 and $13.83 an hour. When the report stacks up the mid-wage jobs with the weakest recovery growth, secretaries and administrative assistants clock in at number two, having lost 345,101 jobs. Third are first-line supervisors and managers of office and administrative workers—those just above the secretaries but in similar roles—who have lost 327,559 jobs. That’s a grand total of 672,660 administrative jobs lost since 2009.

This spells bad news for female workers, who hold three-quarters of these jobs. And the driving force behind the losses is just as troubling. Given that the ability to find a new job is almost nil in the miserable job market, workers are feeling trapped. (For example, the quit rate has hovered at a very low rate since the recovery began.) That gives employers huge leverage in pushing their employees to take on heavier loads without higher pay. This is what Mother Jones calls “the great speed-up”: all work and no pay makes American workers stressed out. In a Spherion Staffing survey, over half of workers had taken on new roles, but just 7 percent had gotten more money for doing so. This trend means that secretaries and administrative assistants are getting the ax, while their workloads get pushed to their remaining coworkers.

This is all, of course, on top of massive public-sector job losses for women, often stemming from widespread teacher layoffs, which also eat away at mid-wage work. And when these unemployed teachers and secretaries find work again, the NELP report shows that the greatest job gains have come in low-wage service jobs like waitresses, retail store clerks and personal and home care aides. Those occupations have added 1.7 million jobs over the past two years—accounting for nearly half of all employment growth.

This trend is happening alongside the loss of blue-collar work: truck drivers are the number-one mid-wage job-losers, and carpenters are number four. Those laid off men may be even more likely to take some of the low-wage jobs, given that they’re making gains in retail, but women are losing jobs there. And women should be concerned that not only are men entering traditionally female sectors during the recovery but when they do they are paid more and meet with better benefits and promotions.

But this isn’t a timed trial in the Oppression Olympics. It’s clear that the chasm in the middle of the job market is hurting women and men alike. While record unemployment levels can’t be explained away by structural issues—if businesses saw more demand, they’d hire more and a flood of relieved unemployed people would be quick to sign up for jobs—the recession and recovery have sped up some ongoing trends. The basis of our economy is quickly shifting from durable, middle-class careers to temporary service-sector work. And the notion that women can take advantage of men’s loss seems troubled at best. The hollowing out of the middle of our labor force means that men and women alike who used to be able to count on good work in the office or the factory are now seeking employment in Walmart and Applebee’s. Grab some moisturizer, because we should all be wringing our hands.