When PEN decided to award the first PEN/Toni and James C. Goodale Freedom of Expression Courage Award to the French satirical magazine Charlie Hebdo, they surely thought they were honoring bravery in defense of free speech. This was a magazine that kept publishing after its offices were firebombed by Islamists in 2011, and kept publishing after nine staffers were horribly murdered by Islamists in January. Compare that to, say, Yale University Press, which dropped the illustrations for Jytte Clausen’s book about the Danish Mohammed cartoons after the book’s first printing, or Random House, which canceled publication of Sherry Jones’s The Jewel of Medina, a historical novel about Mohammed’s wife Aisha. Both publishing houses cited fears of violence by Muslim extremists. Those fears were not irrational. The head of the British publishing house that picked up Jones’s novel had his house firebombed—and the book was dropped. Violence works.

Courage is not enough, though, for the six novelists—Francine Prose, Teju Cole, Michael Ondaatje, Peter Carey, Rachel Kushner, Taiye Selasi—who have withdrawn from participation at PEN’s awards gala next Tuesday. Rachel Kushner criticized the magazine’s “cultural intolerance” and support for “a kind of forced secular view.” Writing in The New Yorker after the murders, Teju Cole condemned Charlie Hebdo for racism and Islamophobia. Peter Carey had trouble with the very concept of freedom of speech being restricted by anything but government. Salman Rushdie promptly called the writers “pussies” on Twitter, which didn’t exactly clarify the issues.

“Charlie Hebdo’s work is not important,” Francine Prose told me, over the phone. “It’s not interesting.” She said she was offended by Charlie’s crude cartoons of the Prophet and mockery of the religion of France’s marginalized Muslim community: “It’s a racist publication. Let’s not beat about the bush.” She compared the magazine’s Muslim caricatures to Goebbels’s anti-Semitic propaganda. “I don’t see a difference, really. It’s the same big noses and thick lips.” She pointed out that many dozens of Mexican and Russian journalists had been killed for reporting on their government’s corrupt doings. Why not honor them?

I’ve known Francine since we were in college, and admire her and her writing enormously. I agree with her that there’s a distinction between supporting the freedom to speak and write, as we both do, and honoring the speech itself. It is probably safe to say that if PEN believed Charlie Hebdo was the Völkischer Beobachter of our day, they wouldn’t be giving it an award, no matter how many of its editors had been massacred. I don’t agree that the drawings of Mohammed are in a different key than the magazine’s rude caricatures of the Pope or Hasidic rabbis or the Virgin Mary just after being raped by the three kings, but maybe that’s in the eye of the beholder. In any case, Charlie is a small satirical magazine run by aging sixties leftists who spend the vast bulk of their column inches attacking the National Front and other French conservatives, with frequent jabs at the Catholic Church. Those immersed in French cartoon culture have pointed out that the offensive drawings circulating on the Internet are, in context, the opposite of what they seem to some American readers—indictments of the racist and anti-immigrant views of right-wing French politicians. In fact, after the murders, Rushdie tweeted that the president of SOS Racisme, the premier anti-racist group in France, had called it “the greatest anti-racist weekly in the country.” Justice Minister Christiane Taubira, whom opponents of the award described as depicted as “a black woman drawn as a monkey” in the pages of Charlie Hebdo, also paid tribute to the magazine.

I don’t agree that Charlie is racist, and not just because Muslims are not a race. Charlie is against all forms of authoritarian religion (Le Monde analyzed ten years of Charlie’s cover stories and found far more attacks on Christianity than on Islam.) Indeed, it is blasphemous. Is that not an honorable left-wing thing to be? It used to be so, before we became so hopelessly confused about Islam: half the time we’re reminding each other that violent fundamentalists like the ones who committed the Charlie Hebdo murders are a tiny fraction of the world’s 1.6 billion Muslims, who are ordinary, nonviolent people of good will, and the other half of the time we talk as if the murderers are out to redress real wrongs—and understandably so, even if the target is poorly chosen. Which is it? I’m not sure that latter view serves Muslims well—it’s a bit like saying people who assassinate abortion providers represent Christians, and West Bank settlers represent Jews.

Garry Trudeau accused Charlie Hebdo of punching down—i.e., making jokes at the expense of the weak and powerless. In a long letter to PEN Executive Director Suzanne Nossel, the short-story writer Deborah Eisenberg said the same thing. There’s something to that—why make struggling people unhappy?—but not enough. It’s basically saying you can make fun of Christianity, but Islam is out of bounds. Furthermore, Charlie doesn’t mock Muslim people—the shopkeeper who runs the corner store, the woman working in a call center, the boys hanging out in the street. It mocks fundamentalism—the narrow, bigoted, superstitious version of Islam that lies behind actually rather a lot of violence against writers. In the West, that violence began with the fatwa against Salman Rushdie for a handful of passages in the Satanic Verses that the Ayatollah Khomeini deemed blasphemous and worthy of death. Many of the writers targeted by fundamentalists have been Muslim themselves, among them the Bangladeshi writer and feminist Taslima Nasreen, the Saudi blogger Raif Badawi condemned by the regime to 1,000 lashes, and, going back a bit in time, the Nobel Prize–winning Egyptian novelist Naguib Mahfouz, thousands of Algerian intellectuals and doubtless many more. These attacks had nothing to do with supposedly racist insults from privileged white people, and everything to do with perceived deviations from orthodoxy.

The six writers are circulating a letter to PEN members, which many great and famous writers are signing: Joyce Carol Oates, Junot Diaz, Lorrie Moore. It seems to me these writers must be awfully sure that they will never fall afoul of either fanaticism or well-meaning liberalism. “There is a critical difference between staunchly supporting expression that violates the acceptable,” it argues, “and enthusiastically rewarding such expression.” Well, sure, but excuse me: violates the acceptable? The acceptable what? And don’t we need writing and artwork that pushes the boundary of what the acceptable is? “The road of excess leads to the palace of wisdom,” as Blake put it.

Norman Mailer, former president of PEN, had all sorts of reprehensible, ignorant and pigheaded views. When it came to politics, he was like a drunken uncle banging the table at Thanksgiving. But he pushed the boundaries back for every writer. And much as I dislike the vast bulk of his writing and his repulsive ideas about women—talk about punching down—if PEN gave him an award, I would just live with it.