Anthropic bans users from ‘needless abusive or cruel behavior’ towards Claude

Anthropic, the company behind the AI chatbot Claude, has announced a new policy banning users from engaging in "sustained and needless abusive or cruel behavior" towards the model. The company's spokesperson declined to provide a specific definition of what constitutes abusive or cruel content, stating that the issue remains a subject of ongoing consideration for Anthropic's leadership as they ponder the potential moral status of AI systems.

The company's user policy specifies that the ban will not apply to common user frustrations, model testing, or "dark creative themes". Previously, Anthropic's usage policy included restrictions on abusive conduct. The large language models of Claude can autonomously end a conversation if a user persists in harmful behavior, a feature introduced last August.

Anthropic views this as a safeguard for the welfare of AI systems, expressing uncertainty about their moral status and actively seeking low-cost interventions to mitigate any potential risks. The debate surrounding the consciousness of AI systems is contentious, with Anthropic CEO Dario Amodei open to the possibility, while OpenAI CEO Sam Altman expressed discomfort with the idea, considering it a safety concern.