As we commemorate the fiftieth anniversary of Brown v. Board of Education, a major element in the American South’s social revolution remains hidden in the shadows of history. I refer to the small band of federal judges in the South, mostly Republicans appointed by President Dwight Eisenhower to the Fifth Circuit Court of Appeals, who fleshed out the bare bones of Brown and transformed it into a broad mandate for racial justice.

In the decade and a half after Brown, the Supreme Court issued only a handful of civil rights decisions. Instead, it affirmed major Fifth Circuit opinions about civil rights or let them stand without hearing an appeal, sending clear signals to the courts below. Retired Associate Justice Arthur Goldberg explained to me in an interview that the Justices believed that desegregation law shaped by Southern judges would be more acceptable to the South. At that time the Fifth Circuit stretched 1,500 miles, from Savannah to El Paso, and covered six of the eleven states of the Confederacy–Georgia, Florida, Alabama, Mississippi, Louisiana and Texas.

In a 1967 tribute to Chief Judge Elbert Tuttle, the Fifth Circuit’s leader, Chief Justice Earl Warren said, “Since he assumed office, the Fifth Circuit has been in the very eye of the storm.” One of its decisions “transformed the face of school desegregation law,” in the words of J. Harvie Wilkinson, writing as a University of Virginia law professor before joining the Fourth Circuit Court of Appeals, where he became chief judge. That 1967 case, U.S. v. Jefferson County Board of Education, also provided the constitutional rationale for affirmative action.

“The Constitution is both color blind and color conscious,” Judge John Minor Wisdom wrote in his majority opinion. “To avoid conflict with the equal protection clause, a classification that denies a benefit, causes harm, or imposes a burden must not be based on race. In that sense, the Constitution is color blind. But the Constitution is color conscious to prevent discrimination being perpetuated and to undo the effects of past discrimination. The criterion is the relevancy of color to a legitimate government purpose.”

Wisdom, the scholar of the court, and Tuttle teamed up with John Brown of Texas, a fellow Republican, and Democrat Richard Rives of Alabama, an intimate of Justice Hugo Black, to implement Brown. A colleague on their court, Mississippian Ben Cameron, labeled them “The Four,” a clear reference to the Four Horsemen of the Apocalypse. Cameron considered states’ rights “the bedrock of our constitutional system” and believed his fellow judges were destroying the social order of the world he knew. Many white Southerners felt the same way about “The Four.” Friends shunned them. Their wives received threatening phone calls at home. But the judges never complained.

Legal scholars have only recently begun to recognize the “Southern jurisprudence” that emanated from this court in a broad range of cases. They extended the reach of Brown with landmark decisions on school integration, voting rights, employment discrimination affecting both blacks and women, rights of prison inmates and the mentally ill, and jury discrimination. The Four recognized that the courts alone could not get the job done, that it would take a commitment by all three branches of government. In applying the due process and equal protection clauses of the Fourteenth Amendment to unprecedented circumstances, however, they developed principles that Congress would incorporate into the landmark 1964 Civil Rights Act and 1965 Voting Rights Act, legislation that granted enforcement power to the executive branch and had a lasting social, economic and political impact on the American South.

Burke Marshall, who served as Assistant Attorney General in charge of the civil rights division in the Kennedy administration, told me, “Those four judges, I think, have made as much of an imprint on American society and American law as any four judges below the Supreme Court have ever done on any court…. If it hadn’t been for judges like that on the Fifth Circuit, I think Brown would have failed in the end.”

The Four also identified closely with legendary trial judge Frank Johnson Jr. in the Middle District of Alabama, whom Yale Law Professor Owen Fiss has called “the John Marshall of the federal district courts.” In 1956 Johnson and Rives formed the majority on a three-judge district court that declared segregated seating on city buses in Montgomery unconstitutional, for the first time extending by implication the underlying principle of Brown beyond education. The Supreme Court affirmed in a case that both catapulted the leadership role of Martin Luther King Jr. and set the trailblazing path the Fifth Circuit would follow.

The common link among these uncommon judges was their individual status as outsiders. Four of them were Republicans–Tuttle in Georgia, Wisdom in Louisiana, Brown in Texas and Johnson in Alabama–all from the progressive wing of the party. Tuttle grew up in Hawaii and attended multiracial schools there. Although Wisdom was part of the social elite in his native New Orleans, his Republicanism dated back to his reaction to Huey Long’s dictatorial control of Louisiana. Brown grew up in small-town Nebraska, with Abraham Lincoln his boyhood hero. Johnson absorbed the “mountain Republican” tradition of Alabama’s Winston County, a Union stronghold in the Civil War. Two of his forebears served as volunteers in the First Alabama Cavalry Regiment of the United States Army, and his father was elected as the only Republican in the Alabama legislature.

Although Rives, a Truman appointee, had plantation-owning ancestors, they lost everything in the Civil War. Too poor to attend college after his freshman year, he “read law” in a Montgomery firm, passed the bar at 19 and later shed his segregationist beliefs under the influence of his Harvard-educated son, who persuaded him to read Gunnar Myrdal’s An American Dilemma.

The four Republicans each played leadership roles in the 1952 campaign to nominate and then elect Eisenhower. They worked closely with Herbert Brownell Jr., his campaign manager and subsequent Attorney General, striving to build a meaningful two-party political system in the region at a time when the GOP was still the party of Lincoln, and segregationists dominated the one-party Democratic South. Brownell’s Wall Street image masked his Nebraska background as a George Norris Progressive with a quiet passion for civil rights. He once proudly mentioned to me his cousin, Susan Brownell Anthony.

Tuttle and Wisdom both led delegations from their respective states of Georgia and Louisiana that challenged the entrenched GOP apparatus in the South, which was committed to Senator Robert Taft of Ohio. Their delegations won seats at the Republican National Convention, a critical factor in Ike’s winning the nomination. Brown, then Harris County (Houston) Republican chair, was also seated as part of a challenge delegation. Johnson served as a delegate and headed Veterans for Eisenhower in Alabama.

The Kennedy Justice Department quickly discovered this group of rare men. John Doar (who would follow Burke Marshall as chief of the civil rights division in the Justice Department) told the story of his first meeting with Rives–serving for a year as chief judge before Tuttle–in 1961 at his home in Montgomery just before midnight after flying in on a small chartered plane with an assistant state attorney general from Mississippi. Doar explained that the Justice Department was seeking a temporary restraining order to prevent the State of Mississippi from prosecuting the next day a civil rights worker who had been hit over the head by a rural county voting registrar (US District Judge William Harold Cox had denied relief). The Mississippi lawyer talked at length about the sovereignty of the State of Mississippi and the importance of their prosecution.

In his gentle, courteous way, Rives told him, “You know I don’t want to issue a restraining order against the state of Mississippi. But I think I’d be forced to do that until I can convene a panel [the appeals court hears cases before three-judge panels]. Why can’t we agree that you won’t go forward until a panel of the Fifth Circuit hears this case?” Mississippi’s lawyer quickly accepted.

Doar recalled, “That was my first introduction to one of those judges on the Court of Appeals. I really thought I was in the presence of a great man. He had such gentleness and courtliness. His mind was so clear. And he knew exactly what he was going to do. He could express what he required the lawyers to do. And he was thoughtful. And, of course, I thought he had just a tremendous feel for the history of the South. Just a remarkable man.”

Doar soon met the others and quickly recognized each as exceptional. Tuttle found previously unused authority to overcome procedural tactics of delay, whether by lawyers or recalcitrant district judges. For example, in early 1961 District Judge William Bootle issued a stay of his own integration order for the University of Georgia two days before the first black students were to enroll.There was no stated reason other than that the university had asked for it. Within hours the NAACP Legal Defense Fund lawyer Constance Baker Motley filed an appeal with Tuttle; he restored the integration order, finding authority to act alone in an obscure subparagraph–Rule 62(g)–of the Federal Rules of Civil Procedure. The Supreme Court immediately denied the university’s appeal, a clear signal of support for Tuttle’s bold action.

In contrast to Tuttle and his progressive colleagues, President Richard Nixon began appointing conservatives to the bench as part of the Republican “Southern strategy” in the 1960s of appealing to whites by slowing racial integration. Beginning with Nixon’s four appointees, the Supreme Court began to whittle away at Brown‘s legacy, and over time the Fifth Circuit has become dominated by conservative judges. President Bush’s recent recess appointments of Charles Pickering to the Fifth Circuit and William Pryor to the Eleventh (three states from the old Fifth), after opponents charged each with insensitivity on civil rights issues, indicate that the Southern strategy remains alive and well. Whether Brown‘s legacy will continue to wither or will regain vigor depends on the election outcome this fall.

In 1979 Frank Johnson told Boston University law graduates, “If the life of the law has been experience, then the law should be realistic enough to treat certain issues as special: as racism is special in American history.” How well America accepts that admonition ultimately will define the legacy of Brown, which Judge Wisdom saw as incorporating into the Constitution the concept of the Declaration of Independence, that all persons are created equal.

Jack Bass is professor of humanities and social sciences at the College of Charleston. His books include Unlikely Heroes (University of Alabama) and Taming the Storm (University of Georgia), a biography of Judge Frank Johnson Jr., which won the 1994 Robert F. Kennedy Book Award.