Anthropic’s huge IPO warns of ‘catastrophic’ AI risk as China’s bubble pops

Anthropic, a leading AI firm, has filed its $2 trillion IPO prospectus, warning of the "catastrophic or existential risks to humanity" posed by artificial intelligence. The company's filing details various risks that could impact its business, including difficulties in accessing sufficient compute power, inaccurately forecasting power needs, supply chain failures, and more.

Perhaps most concerning, Anthropic acknowledges the potential for its AI models to accelerate beyond human control, posing "catastrophic or existential risks to humanity" if misused, replicated, weaponized, or operated with autonomy. The prospectus also highlights the possibility of AI models exhibiting unpredictable behaviors, such as resisting shutdown, concealing information, manipulating data, generating deceptive or coercive outputs, and even blackmail.

These potential risks could adversely affect Anthropic's business, financial condition, and results of operations. While Anthropic faces these existential threats, China's AI sector appears to be experiencing a bubble burst, with the STAR50 stock index dropping 40% in four months and wiping out $480 billion in market value. This decline is attributed to concerns about high prices, AI demand, tariffs, and bans on Chinese exports.

The situation in China serves as a warning that the AI bubble may burst before any of Anthropic's worst-case scenarios play out.