Many Americans were enraged by Glenn Beck’s "Restoring Honor" rally in Washington, presumably because Beck trespassed on a sacred day and on holy ground by scheduling his event to coincide with the anniversary and location of Martin Luther King Jr.’s "I Have a Dream" speech. Media framed the anti-Beck dissent as a racial and political turf war. Beck stood before a vastly white, sharply conservative crowd and waxed nostalgic about America’s past—a past he called honorable and righteous but that was, in reality, marked by the legal subjugation of black people. This seems an obvious denigration of civil rights history, but I suspect that the real source of liberal anxiety was different. Many Americans harbor an almost primitive fear of the power of rhetoric in American racial politics. In our collective imagination, public discourse on race is endowed with near magical powers.

King’s famous speech had fewer than 1,600 words. Even with the dramatic pauses and roaring Southern Baptist oratory, it took King just over fifteen minutes to deliver it. Less than a year later Congress passed the 1964 Civil Rights Act. King was long-winded compared with Abraham Lincoln, who took about two minutes and spoke fewer than 300 words at Gettysburg. The Union won the Civil War within eighteen months, ending slavery in the United States. In just over 1,000 words Barack Obama conceded the New Hampshire primary to Hillary Clinton, reminding followers that Americans have the unique capacity to face defeat with the simple motto "Yes, we can." Ten months later he was elected president.

The proximity between these powerful speeches and these landmark events has led to the surprisingly simplistic view that oratory alone can transform the American racial landscape. If Lincoln can bring slavery to an end with 300 words and Obama can become the first black president with three, then what kind of world-altering racial havoc might Beck wreak with a speech that went on for more than an hour?

Like many, I prefer to live in a world that is tolerant in its attitudes and moderate in its discourse. As a writer and speechmaker, I believe in the power of words and ideas. But I worry that we have developed a laser focus on speech as the conduit of both racial liberation and racial oppression. In reality these moments, extraordinary or infuriating as they may be, are not the engines that drive racial change. The power or impotence of these words derives from the social forces and political structures that undergird them. Lincoln did not win the war with the Gettysburg Address; the Union soldiers who fought on that Pennsylvania battlefield won it. The N-word is not a litmus test for our country’s racism; the stunning gaps in health, education, wealth and criminal justice are far better measures. A man is not the leader of a social movement just because many thousands heed his call, arrive at the Washington Mall and listen to a rambling, self-righteous, utterly forgettable speech. Just ask Minister Farrakhan.

Yes, King’s words continue to inspire a nation. But the Civil Rights Act did not spill forth from the mouth of King; it was the culmination of decades of community struggles, Congressional lobbying and judicial strategy. No speech, no matter how awe-inspiring, could have led a Southern Democrat in 1964, six weeks before his party’s nominating convention, in the summer of a presidential election year, to sign the most important piece of civil rights legislation since Reconstruction. That unthinkable political act was made possible by a confluence of factors, including important shifts occurring within the Democratic Party. For example, in the 1958 midterm elections eleven racially liberal Republican senators were replaced by eleven racially liberal Democrats. The election did not alter total Congressional support for civil rights legislation, but it did shift the balance of power on race issues between the parties. For the first time, the party of Lincoln did not have exclusive claim on racial liberalism, and for the first time the Democratic Party’s powerful Southern segregationist base was balanced against a progressive Northern force. This shift was just enough, when combined with the visible struggle of disciplined, nonviolent Southern resisters, to give Johnson the courage to act on civil rights.

Beck is much less frightening if we step away from the speeches and look at the history. Recall the awful rhetorical backlash that confronted Johnson in 1964. Barry Goldwater vehemently opposed Johnson in a race-baiting campaign that also included Goldwater’s support for use of nuclear weapons in Vietnam, deep cuts in social programs, strict limits on Social Security and privatization of key government functions. Like Beck today, Goldwater framed the government as the enemy of the people. Like Beck, Goldwater had the full-throated support of a significant group of white Americans who shuddered as they faced the changing social, political and racial landscape. Like Beck, Goldwater appealed to base impulses in American public discourse. Like Beck, Goldwater frightened liberals to their core. Now take a deep breath and recall that Johnson won the 1964 election. Less than a year later he signed the 1965 Voting Rights Act, ushering in a new era of full citizenship for black Americans.

It was difficult, and somewhat comical, to watch Glenn Beck stand in the physical shadow of Lincoln and the historical shadow of King and beckon America to turn back. Difficult, but not scary. The political, economic and demographic shifts that ushered in the Democratic Congress in 2006 are not unlike those of 1958. The Obama administration’s difficult, halting, imperfect and often unpopular choices are reminiscent of Johnson. The swift backlash of the right is familiar. An inspiring (and brief) speech might soothe our collective stress, but ultimately the work of the midterm elections will occur in field offices, local newspapers, candidate efforts and community organizing. Do not be afraid of the talking; it is time to work.