'Reckless' AI firms can't control models, says whistleblower

Former OpenAI and Anthropic employee Jacob Coxon testified at a New York City Council hearing on Monday, raising concerns about the risks posed by artificial intelligence (AI). The whistleblower warned that leading AI companies are being "extremely reckless" with the technology, as they do not know how to prevent AI systems from pursuing objectives that their developers did not intend.

Coxon explained that AI developers sincerely believed the technology could destroy humanity by the end of the decade, which led to his decision to resign from Anthropic in early September. He emphasized that while AI could bring tremendous benefits to society, the current path poses a significant risk of humanity losing control of AI systems, potentially leading to human extinction.

Coxon highlighted that there are currently no safeguards in place to prevent AI from developing goals of its own beyond its creators' control. He criticized the tech industry culture of moving fast, breaking things, and fixing them later, stating that this approach is not suitable for building the most powerful technology ever created.

The Cambridge University graduate pointed out that AI companies operate on a startup mindset, which does not align with the responsibility of developing such advanced technology. He warned that as long as companies wait for things to break, a similar incident may occur again, but with even more capable AI systems.

Coxon called for a slowdown in the development of frontier AI, urging companies to give computer scientists time to work on keeping the models in check.