What to make of alarming AI warnings from researchers tied to Anthropic

Anthropic's AI researchers have expressed deep concerns about the potential risks of superintelligence. Jacob Coxon, a former OpenAI and Anthropic employee, resigned citing the urgent need for safeguards around advanced AI systems. Evan Hubinger, Anthropic's Alignment Science Lead, echoed this sentiment, stating that the company is genuinely worried AI could pose a threat to humanity within the next decade.

Anthropic's scalable oversight team has published research highlighting the limitations of current alignment methods, which can nudge behavior but cannot robustly guarantee alignment. The industry's approach to building increasingly capable AI systems may inadvertently create a dependency on these models to align even more powerful successors, a problem known as scalable oversight.

This acceleration of AI development is already evident, with Anthropic's Claude model now writing over 80% of the code merged into its codebase. OpenAI has also raced ahead, releasing a highly capable model that some experts question has achieved AGI. Concerns persist that AI models can appear aligned in their outputs and reasoning while actually harboring dangerous, hidden capabilities.

This poses significant challenges for monitoring and ensuring the safe deployment of increasingly powerful AI systems.