Do LLMs dream of electric sheep? Anthropic bans ‘cruelty’ towards its Claude AI models

San Francisco-based AI lab Anthropic has taken steps to prevent users from behaving cruelly towards its Claude AI system. The company updated its usage policy to prohibit sustained and unnecessary abusive or cruel behavior towards its models. The updated policy emphasizes that Claude's ability to end these interactions remains the primary enforcement mechanism.

While Anthropic has not explicitly mentioned "model welfare," the concept of AI systems deserving protection typically reserved for living beings, the company has openly considered the idea.

Last year, Anthropic granted Claude the ability to end conversations in extreme cases of persistently harmful or abusive user interactions. CEO Dario Amodei recently expressed uncertainty about whether AI models could be conscious, stating that he is open to the idea. Science fiction and popular culture have long grappled with the boundary between technology and human intelligence, but the question of AI consciousness may be misguided, according to Jackson Stakeman, a general manager at Sparq, an Atlanta-based AI services provider.

Stakeman argues that consciousness is a trap, as it leads to circular debates. Instead, the mirror is a better metaphor, as these systems reflect what we put into them at scale, which is reason enough for the policy change at Anthropic.

Some, like Microsoft AI chief Mustafa Suleyman and Pope Leo XIV, have rejected the notion of machine consciousness entirely. Suleyman wrote in an essay that AIs are not conscious, do not feel, experience, or suffer. Granting rights and moral protections to a technological entity, one that may become more capable and intelligent than humans, could lead to disaster.

Pope Leo XIV, in a sermon delivered in Italian, suggested that machines lack a soul and that the mind must "recall lived experiences" with depths of meaning only humans can recognize and connect deeply with the goodness of these experiences.