Anthropic Prohibits “Cruel” Treatment and Abuse of Its Claude AI

As arguments over whether artificial intelligence could ever be conscious intensify, Anthropic has tightened its rules to stop users from subjecting its Claude system to what it calls unnecessary cruelty.

The San Francisco-based AI lab has revised its usage policy to add “a prohibition on sustained and needless abusive or cruel behaviour toward our models.”

“Claude’s ability to end these interactions will remain the primary enforcement mechanism,” the policy states.

While the new language does not explicitly invoke “model welfare”—the notion that advanced AI systems might merit protections typically reserved for living beings—Anthropic and its executives have previously shown a willingness to entertain the broader idea.

Anthropic did not immediately respond to a request for comment.

The company updated its usage policy to include ‘a prohibition’ on abusive or cruel behaviour toward its models

The move follows earlier steps the company has taken to curb abusive use. Last year, Anthropic said it was giving Claude the power to end conversations in rare and extreme situations involving “persistently harmful or abusive user interactions.”

Then in February, CEO Dario Amodei told The New York Times he was uncertain whether AI models could be conscious.

“We don’t know if the models are conscious. But we’re open to the idea that it could be,” Mr Amodei said.

For decades, science fiction and popular culture have fixated on the blurry line between machines and human intelligence. But Jackson Stakeman, a general manager at Sparq, an Atlanta-based AI services provider, argues that focusing on consciousness may miss the point.

“Consciousness is a trap. We can’t prove it in each other. Debate it for AI and you go in circles,” Mr Stakeman told AFP.

Instead, he offered a different frame for understanding why a company might police how people treat chatbots. “The mirror is a better metaphor. These systems reflect what we put in, at scale. That’s reason enough for the policy change” at Anthropic, he added.

Read more:
OpenAI sacks three safety researchers for ‘breach of trust’

Others have dismissed the prospect of machine consciousness altogether, among them Pope Leo XIV and Microsoft AI chief Mustafa Suleyman.

“AIs are not conscious. They do not feel, experience, or suffer,” Mr Suleyman wrote in an essay last month.

He warned that expanding moral protections to machines could backfire. “Granting rights and moral protections to a technological entity, one that looks to be on a path to be seismically more capable and intelligent than us, is a recipe for disaster,” he wrote.

In a sermon delivered in Italian yesterday at St Peter’s Basilica in Vatican City, the pope suggested that machines lack a soul, arguing they simply “compile data” at speed.

“The mind must not simply compile data – as an algorithm now does more quickly than we can,” the pontiff said.

It must “recall lived experiences, which contain depths of meaning that only the human soul can recognise, and connect more deeply with the goodness of these experiences,” the pope said.