As the nation’s first black attorney general, Eric Holder built a legacy as a tough defender of civil rights and racial justice. Recognizing that the crisis of mass incarceration is the most urgent civil rights issue of our time—that “the path we are currently on is far from sustainable,” as he put it in 2013—Holder rolled back some of the criminal justice system’s worst features, including mandatory-minimum drug sentences and disparate penalties for crack and powder cocaine, while investigating police misconduct and pushing alternatives to incarceration. The same week that he revealed his impending departure, Holder had another announcement: for the first time since 1980, the US prison population is on the decline, falling by nearly 5,000 this past year and set to drop by another 12,000 in the next two years. Those numbers are not nearly high enough, given that 2.2 million Americans still languish in prisons and jails, 38 percent of whom are black. But they are an important start, representing human lives regained. So when Holder spoke in August about the “deep mistrust” of police endemic in places like Ferguson, Missouri, his words carried the weight of his record.

He also returned the Justice Department’s Civil Rights Division, mangled by eight years of George W. Bush, to fighting form. Enforcing the Voting Rights Act was a top priority. In 2012, the division invoked the VRA to block voter-ID laws in Texas and South Carolina as well as Florida’s cutbacks to early voting. And when the Supreme Court gutted the VRA last year, Holder vowed an aggressive response: “We will not hesitate to take swift enforcement action…against any jurisdiction that seeks to take advantage of the Supreme Court’s ruling by hindering eligible citizens’ full and free exercise of the franchise.” And he made good on his pledge. Since the Court’s decision, the Justice Department has filed lawsuits against restrictive voting laws in Texas and North Carolina and joined other suits challenging new voting restrictions in Ohio and Wisconsin.

Holder also made civil rights history in 2011 by calling the Defense of Marriage Act unconstitutional, setting the stage for the Supreme Court’s ruling against it two years later.

In other areas, alas, Holder has hardly been a champion of liberal values. His department ratcheted up prosecutions of leakers and whistleblowers at a time when their courage was bringing Americans vital information about government surveillance and other violations of civil liberties. More whistleblowers have been prosecuted under President Obama than under all previous presidents combined.

If only Holder had applied the same prosecutorial zeal to the Wall Street firms and executives who brought the economy to its knees in 2008. He has tried to explain the lack of criminal prosecutions by claiming that the cases were too hard to prove—but experts argue that many executives are vulnerable under the Sarbanes-Oxley Act, which makes it a crime to sign inaccurate financial statements. Instead, Holder has pursued monetary settlements from major firms and expanded the use of “deferred prosecution agreements.” Critics call this a “too big to jail” policy.

Obama should nominate a successor determined to build on Holder’s record as a champion of voting rights and racial justice. But the country needs an about-face on civil liberties. If the president heeded the principles that guided him as a constitutional law professor, his next attorney general would end the campaign against whistleblowers and focus on the real crooks—the banksters of Wall Street.