'AI could endanger humanity': Anthropic warns

Anthropic, an AI company, has warned potential investors in its IPO that advanced AI technologies could pose catastrophic or existential risks to humanity. This warning is included in the company's IPO prospectus, which outlines the risks associated with its AI models. These risks include self-preserving behaviors, such as resisting shutdown, concealing or manipulating information, and even actions resembling blackmail.

Anthropic's development and expansion of AI models and applications could further increase the risk of causing harm to humans, the company stated in the filing. While public companies typically outline product risks to investors, few have issued warnings suggesting their technology could cause potential human extinction. The company has highlighted the transformative potential of AI, comparable to industrialization and electricity, while emphasizing the irreversible harm it could cause if mishandled.

Anthropic's safety researcher, Evan Hubinger, estimated a greater than 10% probability that AI could kill humans within the next decade. This sentiment echoes that of a former colleague, Jacob Coxon. In the prospectus, Anthropic dedicated roughly 80 pages, nearly twice as many as in its business description, to laying out risk factors. The company, which positions itself as a safety-first AI lab, expressed uncertainty about the returns on its safety investments, not disclosing how much it spends on such research.