Attorney General Eric Holder will announce Thursday he is stepping down from the post he has held for nearly six years—making him one of the longest-serving attorneys general in American history.

Holder was the first African-American to hold the position and will surely be remembered as a trailblazer for civil rights. From sentencing reform to combating voter suppression to investigating some of the country’s most violent police forces, Holder made huge progressive strides. It’s no coincidence Holder called the civil rights icon Representative John Lewis on Tuesday before his resignation became public. White House officials are already pushing out narratives about Holder’s “historic legacy of civil rights enforcement and restoring fairness to the criminal justice system.”

But there is one area where Holder falls woefully short: prosecution of Wall Street firms and executives. He came into office just months after widespread fraud and malfeasance in the financial sector brought the American economy to its knees, and yet no executive has faced criminal prosecution. Beyond the crash, Holder established a disturbing pattern of allowing large financial institutions escape culpability.

“His record is really badly blemished by his nearly overwhelming failure to hold corporate criminals accountable,” said Robert Weissman, president of Public Citizen. “Five years later, we can say he did almost nothing to hold the perpetrators of the crisis accountable.”

Advocates for financial accountability often point to the Savings and Loan crisis as a counter-example: despite much smaller-scale fraud, 1,000 bankers were convicted in federal prosecutions and many went to prison.

Holder has tried to explain his lack of prosecutions relating to the 2008 collapse by claiming the cases were too hard to prove—but many experts disagree. The Sarbanes Oxley Act, for example, would provide a straightforward template: it makes it a crime for executives to sign inaccurate financial statements, and there is ample evidence that Wall Street CEOs were aware of the toxicity of the sub-prime mortgages sold by their firms.

Late last year, Judge Jed Rakoff of the Federal District Court of Manhattan wrote an essay in The New York Review of Books bluntly titled, “The Financial Crisis: Why Have No High-Level Executives Been Prosecuted?” He suggested a doctrine of “willful blindness” at Holder’s Justice Department and said “the department’s claim that proving intent in the financial crisis is particularly difficult may strike some as doubtful.” A federal judge will generally not proclaim people guilty outside the courtroom, but Rakoff came close with that statement. The fact he wrote the essay at all stunned many observers.

In recent years, the Justice Department has obtained some large-dollar settlements with Wall Street firms like JPMorgan Chase and Bank of America. But the headline-grabbing amounts end up being significantly less after factoring in tax accounting and credits for actions already being undertaken by the bank. There is also a lack of transparency around how these penalties are being paid to aggrieved consumers.

Holder himself suggested in Senate testimony last year that some firms really are too big to jail:

“I am concerned that the size of some of these institutions becomes so large that it does become difficult for us to prosecute them when we are hit with indications that if you do prosecute, if you do bring a criminal charge, it will have a negative impact on the national economy, perhaps even the world economy,” Holder said.

He later walked that back in subsequent testimony, saying “Let me be very, very, very clear. Banks are not too big to jail.” But the data suggest otherwise.

Under Holder, the Justice Department has greatly expanded the use of deferred prosecution agreements with large corporations, from financial firms to agricultural giants. These are arrangements that take the place of criminal prosecutions—instead, the offending corporation admits wrongdoing, pays a fine (that is usually a small fraction of yearly profits), and agrees to remedy internal problems that lead to the crime. In return, the government agrees not to prosecute.

Public Citizen did an analysis of these agreements at the Department of Justice and found that Holder made them a routine affair:

There isn’t much transparency over which bad actors are awarded deferred prosecution, and which are not, and advocates are alarmed by the precedent.

“[Holder] ensconced the de facto ‘too-big-to-fail’ doctrine by which large financial institutions were effectively immunized form criminal prosecution simply by virtue of being so big,” said Weissman.

This was brought into sharp relief when Justice allowed HSBC to enter deferred prosecution for wide-ranging multibillion-dollar money laundering at the bank on behalf of large illegal drug operations and also terrorist groups.

“The facts are unbelievable. Epic-level money laundering on behalf of narco-traffickers, un-denied by the corporation,” Weissman said. “And no criminal prosecution, and no responsibility for the executives.”

It did seem that Justice became more responsive to these criticisms after Obama was re-elected, and Weissman hopes that continues after Holder. “With a clean slate, Obama should apply a litmus test: a new nominee has to clearly show their willingness to hold accountable the institutions and individuals on Wall Street who devastated Main Street,” he said. “And they have to be crystal clear that they are abandoning the too-big-to-jail doctrine.”