For decades, politicians around the world have repeatedly reminded voters that unauthorized immigrants—or “illegals,” as they like to put it—have broken the law.

By entering the country without permission, staying on after their visas expired, or working outside the official economy, migrants are often doing what the law says they shouldn’t. They are “queue jumpers” who should wait their turn—even though there is no queue they could ever join. They “drain resources” and “steal jobs,” even though they contribute substantially to national economies and work jobs no-one else wants for lower pay. They face a life in the shadows of illegality and continual fear of being separated from those they love, incarcerated and deported. These people have very few chances to state their case.

Because the law is the law and should be obeyed, it’s very easy to agree with the politicians and conclude that migrants have done wrong. But things are more complicated than that. The rule of law requires not just obedience, but also fairness.

We normally think that people should respect and obey the law in democracies, even when we don’t like what it says. That’s because the law provides each of us with a framework in which to live our lives. Though it limits what we can do to pursue our aims, it also restricts what other people can do to us, giving us some security against people and corporations that might act in an abusive or exploitative way.

Of course, that’s just the theory; the law also reflects and sometimes amplifies the inequalities in our societies. That’s why, when people disagree with the law, we presume that they should use democratic means to change it. Such means might include persuading lawmakers, political parties, and their fellow citizens to adopt a different set of policies; voting for parties committed to reform; and perhaps engaging in protest or mild civil disobedience. But foreigners typically can’t do many of those things outside their home country and may risk adverse consequences, including deportation, if they engage in open protest. The democratic process, in other words, does not include them.

Do would-be immigrants, then, actually have a duty to obey immigration laws that tell them they shouldn’t be on the territory, or mustn’t work? One reason to hesitate is that the implicit bargain among citizens, whereby the law limits our freedom in a fair and reciprocal manner, doesn’t really apply to those immigrants. The law coerces them by keeping them out or down—but it doesn’t do anything for them in return. It doesn’t even pretend to treat them fairly.

Some people think this doesn’t matter. They think a country belongs to its citizens and that “we” have a right to pick and choose who can come and who can stay. After all, immigrants have their own countries where they should live their lives unless they have permission to go somewhere else.

But this picture of the world where countries are containers into which people naturally fit is false. People have lives and interests that cross national boundaries. They want to form families or work with others across those lines. As with the Dreamers in the United States, there is often controversy about who should count as part of “we” and where people really belong. Moreover, the opportunities available in different countries are radically unequal: People may need to cross borders just to have minimally decent lives, or to escape persecution or ecological catastrophe.

Does this mean that we should open all borders and allow anyone to go where they want to? Not necessarily. There can be legitimate reasons to limit the numbers of people who want to go to live in a place: Perhaps a country really cannot cope with the strain on its institutions, the economic dislocation, or the environmental stress that large numbers of new people would bring.

But just as it would be tyrannical for one citizen to decide on the law and impose it on everyone else, there is a moral problem with countries just deciding for themselves on immigration and imposing their rules on people who have no say in making them. A fairer system would be made up of international rules for migration, and take account of the interests of both citizens and immigrants, insiders and outsiders, combined with a way a resolving disagreements.

No such system exists, and we need guidance for what to do now rather than waiting for some new international system to emerge. That can’t mean that states just get to do whatever they like in the meantime. Rather, to have the kind of authority that the law claims, states have to commit to working with other states and with migrants themselves to develop a fairer system. And they have to uphold those existing treaties, such as the Refugee Convention of 1951, that many states are currently undermining.

They also must anticipate, in their current immigration policies, what a fair system would look like: by refraining from separating families, or detaining people fleeing for their lives, and which give more people a shot at economic opportunity.

The rule of law isn’t just about people obeying the law. It is about having a fair system that works for everyone. A system where some states get to impose their will on outsiders who get neither a chance to shape the law, nor the opportunity to defend themselves against incarceration or deportation, isn’t an example of law in action but of naked force. And people don’t have a duty to submit to naked force: they have a right to resist it.

If states want the right and not just the power to “control their borders,” then they have to take account of the rights and interests of immigrants. Otherwise, there is no law—only tyranny.