Anthropic warns AI may pose 'existential risks to humanity' in IPO filing

Anthropic, an AI company, has warned in its IPO filing that advanced AI technology could pose catastrophic or existential risks to humanity. The company's prospectus, reviewed by Reuters, outlines potential risks associated with its AI models, including self-preservation behaviors such as resisting shutdown, concealing information, and manipulating data.

Anthropic believes that the development and expansion of its AI models could further increase the risk of harm. While public companies usually disclose product risks, Anthropic's warning is rare and suggests a potential for human extinction. The company has emphasized the transformative potential of AI, comparable to industrialization and electricity, but also the irreversible harm it could cause if mishandled.

Safety researcher Evan Hubinger estimates a greater than 10 percent probability that AI could kill humans within the next decade. Anthropic, which positions itself as a safety-first AI lab, has dedicated nearly twice as much space in its 261-page prospectus to risk factors compared to SpaceX's 277-page document. The company has not disclosed how much it spends on safety research, but admitted that it is a resource-intensive aspect of their operations.

Despite the emphasis on safety, Anthropic acknowledges unclear returns on their safety investments and has pledged to disclose more data publicly about their AI model development.