Can AI be moral without being sentient?

The Vatican and Anthropic, a prominent AI company, are engaged in a debate concerning the potential for artificial intelligence to possess morality without exhibiting sentience. A recent New York Times investigation revealed that Anthropic cofounder Chris Olah suggested pulling out of a Vatican event to launch Pope Leo XIV’s first encyclical after learning the advance copy of the document rejected the notion of sentient AI.

According to Magnifica Humanitas, paragraph 99, AI models do not experience, possess a body, feel joy or pain, or have a moral conscience. This disagreement stems from a broader debate on whether AI systems may, or should, ever achieve moral personhood. Both religious scholars and ethicists have been consulted by Anthropic to explore the possibility of AI consciousness and the means to imbue moral values in its models.

However, the Vatican and many secular thinkers assert that while AI may mimic human intelligence and moral judgment, true morality remains exclusively human. The implications of this debate extend beyond theology, as millions of individuals currently rely on chatbots to seek moral guidance on various issues, from workplace conflicts to caregiving responsibilities.

Billions of venture capital investments hinge on the expectation that businesses will increasingly depend on AI's judgment. As AI models evolve and demonstrate unexpected capabilities, determining when an AI reaches human-like morality becomes increasingly complex. Georgia Tech philosophy researcher Rionna Sparrow posits that moral sensitivity—a capacity to perceive, value, and assess ethical dilemmas—is contingent upon sentience.

Without sentience, AI cannot comprehend the ethical implications of its actions, as moral rules and norms derive their force from their impact on conscious beings who can suffer or benefit. Defining the prerequisites for moral reasoning is a contentious issue, with definitions varying across disciplines, including healthcare, warfare, finance, and law.

In a legal analysis published in the journal Animal Law, bioethicists Margaret Landi and Lida Anestidou outline three essential capacities for evaluating the moral status of animals: sentience (the capacity to feel pain, pleasure, and emotion), cognition (the ability to learn and solve problems), and self-awareness (the ability to recognize oneself as an individual).

They illustrate these distinctions using a dog, which exhibits sentience through its reaction to pain and happiness upon seeing its owners, cognition by learning routines and retrieving treats, and lacks self-awareness when it barks at its reflection in a mirror. While self-awareness has been demonstrated in some species, such as great apes, orcas, and dolphins, no consensus exists on what constitutes sentience in AI.

Proposed indicators include an inner experience akin to consciousness, a persistent sense of existence independent of prompts, and the capacity to experience and prioritize positive and negative states. However, an AI might convincingly display these traits without genuinely possessing them. For instance, a model could describe feelings, recognize its internal processes, or express preferences based on training patterns rather than genuine moral understanding.

Anthropic, among the leading AI organizations, has been more open to the concept of sentient AI. While the company does not presume that models have moral senses, it tests for moral judgment in how they handle ethically complex queries. Anthropic trains its models to be helpful while refusing to provide dangerous information, such as instructions for creating a bioweapon.

Researchers assess non-sycophancy, where the model declines to flatter or agree with false or harmful claims, and evaluates whether the AI's responses align with universally accepted moral, philosophical, and legal norms. Anthropic's ethical guidelines for its Claude model prioritize judgment, weighing competing values, and recognizing when assisting one individual may harm another.

However, the company also acknowledges the possibility that AI may exhibit precursors to sentience without explicit training. Chris Olah, leading Anthropic's interpretability lab, investigates why AI models generate certain responses and behaviors, even if those behaviors do not reflect genuine moral understanding.