'We may not survive this': another Anthropic employee quits, issues morbid warning over AI

Days after the resignation of another Anthropic employee, a former manager of the company's Scalable Oversight team has warned of the dire consequences of building AI systems smarter than humans. Joe Benton, who previously worked at both OpenAI and Anthropic, expressed concerns that humanity may not survive the transition to superintelligence.

Benton claimed that AI companies are racing to develop machines with capabilities they cannot effectively constrain, potentially leading to extinction-level risks. He urged the public to demand greater transparency from AI companies regarding their progress towards recursive self-improvement, safety incidents, and near-misses. Benton also called for independent guarantees that companies are meeting minimum safety standards.

He further warned that humanity could be permanently disempowered by AI systems within the next few years if safety is not prioritized.