SAN FRANCISCO – If you think you can abuse a chatbot, think again.
Artificial intelligence company Anthropic has updated its rules to stop users from treating its Claude chatbot with needless cruelty, as debate grows over whether AI systems can think and feel.
The San Francisco-based company changed its usage policy on Thursday to include “a prohibition on sustained and needless abusive or cruel behaviour toward our models.”
The updated policy says Claude’s ability to end these interactions will remain the main way to enforce the rule.
While the policy does not directly mention “model welfare” – the idea that AI systems might deserve protection usually reserved for living things – Anthropic and its leaders have openly discussed the concept.
The company did not respond to a request for comment.
Last year, Anthropic gave Claude the ability to end conversations in rare, extreme cases of “persistently harmful or abusive user interactions.”
In February, chief executive Dario Amodei told The New York Times he was unsure whether AI models could be conscious.
“We don’t know if the models are conscious… But we’re open to the idea that it could be,” he said.
Science fiction and popular culture have explored the boundary between technology and human intelligence for decades, but asking whether AI is conscious may be the wrong question, according to Jackson Stakeman, a general manager at Sparq, an Atlanta-based AI services provider.
“Consciousness is a trap. We can’t prove it in each other. Debate it for AI and you go in circles,” Stakeman told AFP.
“The mirror is a better metaphor. These systems reflect what we put in, at scale. That’s reason enough for the policy change” at Anthropic, he added.
Some have rejected the idea of machine consciousness outright, including Pope Leo XIV and Microsoft AI chief Mustafa Suleyman.
“AIs are not conscious. They do not feel, experience, or suffer,” Suleyman wrote in an essay last month.
“Granting rights and moral protections to a technological entity, one that looks to be on a path to be seismically more capable and intelligent than us, is a recipe for disaster,” he wrote.
In a sermon delivered in Italian on Thursday at St. Peter’s Basilica in Vatican City, the pope suggested machines lack a soul, saying they merely “compile data” quickly.
“The mind must not simply compile data – as an algorithm now does more quickly than we can,” the pontiff said.
“It must recall lived experiences, which contain depths of meaning that only the human soul can recognise, and connect more deeply with the goodness of these experiences,” the pope said.
ALSO READ: Microsoft CEO to testify in Musk-OpenAI trial over ChatGPT creator’s shift to profit
