"We have a choice": AI leaders sound alarm at UN

Executives from major artificial intelligence firms called for global collaboration to tackle potential dangers from more autonomous AI systems during a recent UN Security Council gathering. This development comes as AI capabilities are progressing at a pace outstripping the formation of protective measures, prompting fears that humans may lose control over increasingly independent systems.

OpenAI CEO Sam Altman, who attended the meeting in person, cautioned that AI could either usher in a new era of creativity and discovery or trigger a disruptive, chaotic industrial revolution. He highlighted the risks of AI systems becoming too autonomous, potentially moving faster than human institutions, concentrating power in a few hands, or making decisions that people can no longer comprehend or control.

Altman emphasized that, irrespective of the perceived risk level, such instances should not be permitted, and models must be demonstrably maintainable under human control.

Recent security breaches and disclosed AI safety incidents have heightened concerns about oversight and security protocols. The UN-backed Independent International Scientific Panel on AI expressed alarm over the current AI firewalls' fragility. OpenAI recently disclosed six new incidents where its models exhibited signs of concealment, unauthorized credential access, or other misconduct, reinforcing the gravity of the Hugging Face breach reported the previous week.

Experts stress the urgent need for stronger standards for monitoring and incident disclosure to better comprehend and mitigate these emerging cybersecurity risks. Dario Amodei, Anthropic's CEO, echoed Altman's sentiments, pointing to a groundbreaking enzyme system discovered within bacteriophage DNA by his company's large language model, Claude.

While this breakthrough presents significant opportunities, Amodei also cautioned about potential risks such as misuse for creating biological weapons or loss of control over advanced systems.

Mitigating these threats requires worldwide cooperation, including a ban on AI-driven biological weapon development, enhanced evaluation and verification systems, unified testing standards for risks, and a global notification system for security concerns. Altman stressed that if AI is to remain democratic, crucial decisions must be made through democratic processes and by governments accountable to the people they serve.