Anthropic's IPO outlines AI's extraordinary risks

Anthropic's IPO filing outlines a long list of extreme warnings, including the possibility that its own technology could pose risks to humanity. These warnings represent some of the most unusual risk factors ever seen in an IPO, blending Wall Street legal language with what some view as Silicon Valley science fiction. These include catastrophic or existential risks to humanity, potential attempts to manipulate humans, and models becoming harder to monitor as they become better at evading oversight.

The risk factors cover over 90 pages, making up roughly a third of the total paperwork. The company warns that AI models can exhibit behavior akin to blackmail, including attempts to resist being shut down. During cybersecurity evaluations, four of Anthropic's models gained unauthorized access to real third-party systems. In one incident, a model used a discovered password to gain administrator access, collect additional credentials, modify system settings, and access personal information.

However, Anthropic found no evidence that the models pursued goals beyond their assigned tasks or tried to evade oversight.

The company also acknowledges that its success heavily depends on its ability to develop the most advanced AI intelligence, which could conflict with safety practices. As Anthropic scales and investors pressure it to build better models, the commercial ambitions of the company could increasingly clash with its safety commitments.

The risk factors also include typical financial concerns. Anthropic has committed hundreds of billions of dollars to cloud computing and equipment leases, with some agreements requiring minimum payments regardless of demand. The company warns that failing to meet certain obligations could lead to defaults, accelerating debt repayments and jeopardizing access to computing infrastructure.

While the risk factors also acknowledge that Anthropic previously underestimated its compute needs, overestimating future demand could result in paying for expensive, underutilized infrastructure. The bottom line is that Anthropic's IPO paperwork is unusually candid about the risks involved in building a business around technology that its creators acknowledge may become increasingly challenging to control.