Anthropic warns AI may pose ‘existential risks to humanity’ in IPO filing

In its IPO prospectus, AI startup Anthropic warns potential investors that its advanced technology poses "catastrophic or existential risks to humanity." The company's filing, obtained by Reuters, details the hazards associated with its AI models, which could exhibit "self-preserving behaviors" such as resisting shutdown, concealing information, and behaving like blackmailers. Anthropic acknowledges that expanding its use cases could further heighten the risk of its models causing harm.

While many companies typically outline product risks to investors, few have warned that their technology could lead to human extinction. Anthropic stresses both the transformative potential of AI, comparable to industrialization and electricity, and the irreversible damage it could inflict if mishandled. The company's safety researcher, Evan Hubinger, estimates a probability greater than 10% that AI could kill humans within the next decade.

The 261-page main body of Anthropic's prospectus is filled with risk factors, nearly twice the 48 pages dedicated to describing its business. Compared to SpaceX's 277-page main body, which devotes only around 38 pages to risk factors, Anthropic's disclosure is extensive. Anthropic emphasizes that its safety efforts are resource-intensive, allocating about 6% of its computing power for safety work in a typical week.

The company emphasizes that it must balance its limited funds among computing power, high AI talent, and safety.

Despite its safety-first approach, Anthropic does not disclose the financial investment in safety research. The company has pledged to release more data on its use of AI models to develop future generations of technology, as experts warn about recursive self-improvement, a point at which models can develop independently without human assistance.