Why AI researchers keep building something they think will kill humans

AI researchers continue to develop what they fear could lead to humanity's demise, driven by a mix of commercial incentives and a belief that they are in a race against less responsible AI developers. For OpenAI and Anthropic, the motivation to create more powerful AI systems is rooted in the potential for significant financial gain.

Employees at these companies feel compelled to push forward with their work, even if they privately express concerns about the potential dangers of their creations. Additionally, some researchers believe they are uniquely positioned to ensure the safe development of AI technology, leading them to continue their efforts despite the risks.

The intense focus on safety and alignment within these organizations may contribute to the growing sense of panic surrounding the potential for AI to cause catastrophic harm. As AI systems become increasingly powerful and autonomous, researchers worry about the loss of agency in the development process. With control increasingly concentrated in the hands of a few major labs, researchers feel they have little influence over AI's trajectory, which can fuel anxiety and lead to dire predictions about an impending AI-driven apocalypse.