Last week, the attorney Jon Eisenberg was trying to get in touch with a client of his—a Syrian named Jihad Ahmed Mujstafa Diyab, who has been detained at Guantánamo Bay for nearly twelve years without trial. Twice a day for months, medical staff and guards have dragged Diyab from his cell, strapped him in a chair, and forced a tube down his throat, ostensibly to ensure that he doesn’t die during a prolonged hunger strike.

Eisenberg had trouble reaching Diyab. All telephone communication between detainees and their lawyers had been shut down, Eisenberg says—presumably because President Obama was in the midst of delicate and secret negotiations with the Taliban over exchanging five other men held at Guantánamo for an American prisoner of war named Bowe Bergdahl. (A spokesperson for Joint Task Force Guantanamo said he could not confirm or deny Eisenberg’s account.)

The five Guantánamo detainees, reportedly Taliban officials, have now been escorted to Qatar, where they will be able to walk freely on the streets. Left at the military prison are seventy-eight other men who have been cleared for release yet remain in interminable detention. Most, like Diyab, were cleared more than four years ago. Though Obama continues to profess his commitment to closing the prison, and though constraints placed by Congress on his ability to do so have weakened considerably, his administration has not demonstrated the resolve to transfer the remaining prisoners.

“From the administration’s perspective, I think they must have seen more urgency from Bergdahl,” said Eisenberg. “From my client’s perspective, it’s plenty urgent. He’s being tortured every day.”

Just weeks ago the president of Uruguay, Jose Mujica Cordano, offered to accept six of the cleared men as refugees. Secretary of Defense Chuck Hagel said on May 28 that he would decide to reject or accept the proposal “fairly soon.” Now that Obama has shown a willingness to push legal boundaries in order to move detainees whose designation as a threat seems at least plausible, the circumstance of men like Diyab, who the government never intends to charge with a crime, is even more indefensible.

“What’s changed is that the president has finally taken the initiative,” said Wells Dixon, a senior attorney at the Center for Constitutional Rights, which represents eight Guantánamo detainees. “What we’re hoping is that he will continue to take bold steps to transfer the remaining men, who are not nearly as complicated or as controversial.”

The administration’s legal authority to move the men who’ve been cleared for release is much clearer than it appears in the Bergdahl swap. According to the National Defense Authorization Act passed last year, the secretary of defense needs only to notify Congress of any prisoner transfers thirty days beforehand.

The most significant complication is “the Yemen problem.” Yemenis make of the bulk of the men who’ve been cleared for release, but none has been repatriated since 2009. Obama announced more than a year ago that he would lift a ban on repatriating Guantánamo’s Yemeni prisoners, a moratorium he himself imposed after a man who’d allegedly trained with an Al Qaeda affiliate in Yemen tried blow up an airplane bound for Detroit using bomb hidden in his underwear. US and Yemeni officials have discussed a plan to move prisoners to a yet-to-be-built detention center outside the capital Sana’a, but the proposal faces funding issues, and there are outstanding questions about whether it would function as a rehabilitation center or merely another prison.

“That is the biggest barrier, figuring out where to send the Yemenis,” said Andrea Prasow, who works on detainee issues for Human Rights Watch. From her perspective, concerns about safety don’t justify indefinite detention. “There’s a possibility that anyone on the streets in Yemen—or anywhere—can pose a risk to us, but we don’t lock those people up,” she said. “There’s only one way to close Guantánamo, and that’s by sending people home.”

Prasow is hopeful that the Bergdahl deal “signifies a sea change” in the president’s willingness to act swiftly to relocate the remaining Guantánamo detainees. “There’s all this political backlash, but the administration is going to weather it. Meanwhile there’s tremendous backlash around the world about the fact that Guantánamo is still open.”

Eisenberg, who represents three other prisoners along with Diyab, is less optimistic. “You would think that [transferring the cleared detainees] would be the easier decision for Hagel to make,” he said. “There are different pressures brought to bear in the Bergdahl case.”

Ultimately, the Bergdahl case illustrates that the greatest barriers to closing Guantánamo are political, not legal or logistical, and that they remain in place only so long as the president allows. Republicans who would like to keep Guantánamo open will no doubt channel the right-wing outrage over the release of the five Taliban prisoners to their own ends. But their bluster has no bearing on the president’s authority to send home men whose freedom is both recommended by US officials, and long overdue. There’s simply no way to close Guantánamo without doing so.