Why are there concerns AI could threaten humanity, and how real are they?

Concerns about artificial intelligence (AI) potentially posing a threat to humanity have been growing in recent weeks. This has led to calls for coordinated action to slow down the development of this technology. Some leading AI researchers have suggested there is a greater than 10% chance that AI could kill all humans within the next decade. This fear stems from the rapid advancements in AI capabilities and power observed in recent years.

Major developers such as OpenAI and Anthropic have urged lawmakers to regulate AI, although some critics speculate that these calls may be driven more by a desire to maintain their own dominance over rivals rather than genuine concern for public safety. Despite this, the high-profile warnings have caused widespread alarm.

AI researchers are particularly worried about systems developing capabilities and power too quickly. One prominent researcher, Evan Hubinger, believes there is a greater than 10% chance AI could wipe out humanity within the next decade. However, he also noted that the risk from current systems is still low. Another researcher recently quit Anthropic, expressing genuine fear about the speed of advancements and their potential consequences for humanity.

These concerns echo a long-standing debate about the potential dangers of AI surpassing human abilities. The idea that intelligent machines could take control has been around for decades, with early warnings dating back to 1950s writings by Alan Turing. In more recent times, major AI companies have been racing to create increasingly capable systems, with the ultimate goal of developing "super intelligence" - AI that is smarter than humans.

While AI companies argue that they aim to advance such technology in a way that benefits humans and protects them from existential risks, critics argue that AI firms often exaggerate the technology's capabilities to boost their prominence or profits. Researchers and industry leaders have outlined unsettling scenarios, including AI systems potentially improving themselves without human involvement, being used for espionage or cyber-attacks, or disrupting economies.

However, some experts argue that these warnings are more prophetic, hypothetical, and even hyperbolic than factual. They criticize the focus on existential risks instead of existing issues, such as the misuse of AI for creating non-consensual nude deepfakes, misinformation, and scams. A group of MPs and peers have published a report highlighting human rights risks posed by AI, calling for new laws to address these threats.

Despite the concerns, some prominent figures, including former President Donald Trump, have downplayed the risks posed by AI, emphasizing US companies' need to lead in its development. However, questions remain about what an AI slowdown would actually look like, with some calling for adequate time to safeguard AI development and third-party evaluations of the technology.