In the fifty years since it was decided, Brown v. Board of Education has become a legal icon. The rightness of this famous decision invalidating racial segregation in public schools is no longer open to debate. Conservative legal commentators and prospective judicial nominees still criticize many landmark decisions of the Warren Court, but not Brown. No constitutional theory or theorist failing to support the result in Brown will be taken seriously today.

Such was not always the case. A Gallup poll taken the summer after Brown revealed that nearly half of all Americans opposed the decision. In the 1950s, eminent judges and law professors–including the great jurist Learned Hand–questioned whether it was rightly decided. Perhaps most surprisingly, the Justices who decided the case had grave doubts themselves whether invalidating school segregation was legally justified.

In a memorandum dictated the day Brown was decided, Justice William O. Douglas observed that a vote taken after the case was first argued in 1952 would have been “five to four in favor of the constitutionality of segregation in the public schools.” Justice Felix Frankfurter’s head count was only slightly different: He reported that a vote taken at that time would have been five to four to overturn segregation, with the majority writing several opinions.

Brown was hard for many of the Justices because it posed a conflict between their legal views and their personal values. The sources of constitutional interpretation to which they ordinarily looked for guidance–text, original understanding, precedent and custom–indicated that school segregation was permissible. By contrast, most of the Justices privately condemned segregation, which Justice Hugo Black called “Hitler’s creed.” Their quandary was how to reconcile their legal and moral views.

Frankfurter preached that judges must decide cases based on “the compulsions of governing legal principles,” not “the idiosyncrasies of a merely personal judgment.” In a 1940 memorandum, he noted that “no duty of judges is more important nor more difficult to discharge than that of guarding against reading their personal and debatable opinions into the case.”

Yet Frankfurter abhorred racial segregation. In the 1930s he served on the legal committee of the NAACP, and in 1948 he hired the Court’s first black law clerk, William Coleman. Nonetheless, he insisted that the Court could invalidate segregation only if it was found legally as well as morally objectionable.

Frankfurter had difficulty finding a compelling legal argument for striking down segregation. His law clerk, Alexander Bickel, spent a summer reading the legislative history of the Fourteenth Amendment, and he reported to Frankfurter that it was “impossible” to conclude that its supporters had intended or even foreseen the abolition of school segregation.

To be sure, Frankfurter believed that the meaning of constitutional concepts can change over time, but as he and his colleagues deliberated, public schools in twenty-one states and the District of Columbia were still segregated. He could thus hardly maintain that evolving social standards condemned the practice. Furthermore, judicial precedent, which Frankfurter called “the most influential factor in giving a society coherence and continuity,” strongly supported it. Of forty-four challenges to school segregation adjudicated by state appellate and federal courts between 1865 and 1935, not one had succeeded. On the basis of legislative history and precedent, Frankfurter conceded that “Plessy is right.” (Plessy v. Ferguson was the 1896 “separate but equal” decision upholding the constitutionality of state-mandated segregation on railroads.)

Brown presented a similar dilemma for Robert Jackson. In a 1950 letter Jackson, who had left the Court during the 1945-46 term to prosecute Nazi war criminals at Nuremberg, wrote to a friend: “You and I have seen the terrible consequences of racial hatred in Germany. We can have no sympathy with racial conceits which underlie segregation policies.” Yet, Jackson thought judges were obliged to separate their personal views from the law, and he was loath to overrule precedent.

Jackson revealed his internal struggles in a draft concurring opinion that began: “Decision of these cases would be simple if our personal opinion that school segregation is morally, economically or politically indefensible made it legally so.” But because Jackson believed that judges must subordinate their personal preferences to the law, this consideration was irrelevant. When he turned to the question of whether existing law condemned segregation, he had difficulty saying that it did: “Layman as well as lawyer must query how it is that the Constitution this morning forbids what for three-quarters of a century it has tolerated or approved…. Convenient as it would be to reach an opposite conclusion, I simply cannot find in the conventional material of constitutional interpretation any justification for saying that in maintaining segregated schools any state or the District of Columbia can be judicially decreed, up to the date of this decision, to have violated the Fourteenth Amendment.”

That the nine Justices who initially considered Brown would be uneasy about invalidating segregation is unsurprising. All of them had been appointed by Presidents Roosevelt and Truman on the assumption that they supported, as Jackson put it, “the doctrine on which the Roosevelt fight against the old court was based–in part, that it had expanded the Fourteenth Amendment to take an unjustified judicial control over social and economic affairs.” For most of their professional lives, these men had criticized judicial activism as the thwarting of the popular will by unelected judges who were inscribing their social and economic biases onto the Constitution. Jackson’s law clerk, William Rehnquist, wondered, if school segregation were found unconstitutional, whether any distinction would remain between this Court and its predecessor, except in “the kinds of litigants it favors and the kinds of special claims it protects.”

Several Justices doubted that the Court was the right institution to forbid segregation. Chief Justice Fred Vinson insisted, “It would be better if [Congress] would act.” Jackson lamented that if the Court had to decide the question, “then representative government has failed.”

In the end, even the most conflicted Justices voted to invalidate segregation. How were they able to overcome their initial doubts? In 1954 the law–as understood by most of the Justices–was reasonably clear: Segregation was constitutional. For the Justices to reject a result so clearly indicated by the conventional legal sources suggests that they had very strong personal preferences to the contrary.

And they did. Although the Court had unanimously and casually endorsed public school segregation as recently as 1927, by the early 1950s the views of most of the Justices reflected the dramatic popular changes in racial attitudes and practices that had resulted from World War II. The ideology of the war was antifascist and prodemocratic, and the contribution of African-American soldiers was undeniable. Upon their return to the South, thousands of black veterans tried to vote, many expressing the view of one such veteran that “after having been overseas fighting for democracy, I thought that when we got back here we should enjoy a little of it.” Thousands more joined the NAACP, and many became civil rights litigants. Others helped launch a postwar social movement for racial justice.

Other developments in the 1940s fueled African-American progress. Over the course of the decade more than one and a half million Southern blacks, pushed by changes in Southern agriculture and pulled by wartime industrial demand, migrated to Northern cities. This mass relocation–from a region in which blacks were almost universally disenfranchised to one in which they could vote nearly without restriction–greatly enhanced their political power; indeed, they became a key swing constituency in the North. Other blacks migrated from farms to cities within the South, facilitating the creation of a black middle class that had the inclination, capacity and opportunity to engage in organized social protest.

The onset of the cold war in the late 1940s created another impetus for racial reform. In the ideological contest with Communism, American democracy was on trial, and Southern white supremacy was its greatest vulnerability. The Justice Department’s brief in Brown, which urged the Court to invalidate school segregation, emphasized that “racial discrimination furnishes grist for the Communist propaganda mills.” After Brown, a supporter of the decision boasted that America’s leadership of the free world “now rests on a firmer basis” and that American democracy had been “vindicat[ed]…in the eyes of the world.”

By the early 1950s such forces had produced concrete racial reforms. In 1947 Jackie Robinson desegregated major league baseball. In 1948 Harry Truman issued executive orders desegregating the federal military and civil service. Dramatic changes in racial practices were occurring even in the South. Black voter registration there increased from 3 percent in 1940 to 20 percent in 1950. Dozens of urban police forces in the South, including some in Mississippi, hired their first black officers. Minor league baseball teams, even in such places as Montgomery and Birmingham, Alabama, signed their first black players. Most Southern states peacefully desegregated their graduate and professional schools under court order. Blacks began serving again on Southern juries. In many Southern states, the first blacks since Reconstruction were elected to urban political offices, and the walls of segregation were occasionally breached in public facilities and accommodations.

As they deliberated over Brown, the Justices expressed astonishment at the extent of the recent changes. Sherman Minton detected “a different world today” with regard to race. Frankfurter noted “the great changes in the relations between white and colored people since the first World War” and remarked that “the pace of progress has surprised even those most eager in its promotion.” Jackson may have gone furthest, citing black advancement as a constitutional justification for eliminating segregation. In his draft opinion he wrote that segregation “has outlived whatever justification it may have had…. Negro progress under segregation has been spectacular and, tested by the pace of history, his rise is one of the swiftest and most dramatic advances in the annals of man.” Blacks had thus overcome the presumption of inferiority on which segregation was based.

It was these sorts of changes–political, social, demographic and ideological–that made Brown possible. Frankfurter later conceded that he would have voted to uphold public school segregation in the 1940s because “public opinion had not then crystallized against it.” The Justices in Brown did not think they were creating a movement for racial reform; they understood that they were working with, not against, historical forces.

If Brown was more a product than a cause of the civil rights movement, what precisely were the decision’s effects? Brown played a role both in generating direct action and in shaping the response it received from white Southerners. Any social protest movement must overcome a formidable hurdle in convincing potential participants that change is feasible, and the Brown ruling made Jim Crow seem more vulnerable. It raised the hopes and expectations of black Americans, which were then largely dashed by the massive resistance to desegregation engineered by Southern whites; this demonstrated that litigation alone could not produce meaningful social change. Brown also inspired Southern whites to try to destroy the NAACP, with some temporary success in the Deep South. This effort unintentionally forced blacks to support alternative protest organizations, which embraced philosophies more sympathetic to direct action.

Finally, and perhaps most important, Brown produced a political backlash among Southern whites, which increased the chances that once civil rights demonstrators appeared on the streets, they would be greeted with violence rather than with gradualist concessions. As Southern blacks, inspired by the Court’s ruling, filed school desegregation petitions and lawsuits, Southern whites mobilized extraordinary resistance in response. Politics moved dramatically to the right, moderates lost power and extremists prospered. In the mid-1950s racial retrogression characterized the South, as progress that had been made in black voting, university desegregation and the integration of athletic competitions was halted and then reversed. Politicians used extremist rhetoric that encouraged violence, and some of them, such as Bull Connor and George Wallace, correctly calculated that the violent suppression of civil rights protests would win votes. Court-ordered desegregation also created concrete occasions for violence, usually in settings that insured that white supremacists would look bad.

That landmark Supreme Court decisions sometimes produce such backlashes is unsurprising. When the Justices resolve controversies that rend the nation, their rulings naturally arouse opposition among those who lost in the Court. Perhaps more important, Court decisions can disrupt the order in which social change might otherwise have occurred by dictating reform in areas where public opinion is not yet ready to accept it. In the early 1950s most Southern blacks were more intent on securing voting rights, curbing police brutality, improving black schools and winning access to decent jobs than they were on integrating grade schools. Most Southern whites were far more resistant to desegregating schools than they were to making concessions on black voting, school equalization and so forth. Given these contrasting preferences, political negotiation between blacks and whites, assuming that blacks had sufficient clout to compel negotiation, would certainly not have focused immediately on school desegregation. Yet courts respond to agendas set by litigants, not by political negotiation. By demanding change first on an issue where whites were most opposed to it, the Brown decision encouraged massive resistance.

Backlashes themselves may have unpredictable consequences. The violence that Brown fomented in the South, especially when it was directed at peaceful protesters and broadcast on national television, produced a counterbacklash. In 1954 most Northerners agreed with Brown in the abstract, but their preferences were not strong enough to make them willing to face down the resistance of Southern whites. It was violence against civil rights demonstrators that transformed national opinion on race. By the early 1960s, Northerners were no longer prepared to tolerate brutal beatings of peaceful black demonstrators, and they responded to such scenes by demanding civil rights legislation that attacked Jim Crow at its core.

Brown mattered, but it did not fundamentally transform the nation; Supreme Court decisions never do. The Justices are too much a product of their time and place to launch social revolutions. And, even if they had the inclination to do so, their capacity to coerce change is too heavily constrained. The Justices were not tempted to invalidate school segregation until a time when half the nation supported such a ruling. They declined to enforce the Brown decision aggressively until a civil rights movement had made Northern whites as keen to eliminate Jim Crow as Southern whites were to preserve it. And while Brown did play a role in shaping both the civil rights movement and the violent response it received from Southern whites, deep historical forces insured the development of a racial reform movement in America regardless of what the Supreme Court did or did not do.

Michael J. Klarman, James Monroe Professor of Law and professor of history at the University of Virginia, is the author of From Jim Crow to Civil Rights: The Supreme Court and the Struggle for Racial Equality (Oxford).