Open-source AI like 'a Nazi coach', UN-backed anti-terror group says

An Australian teen, 13 years old, allegedly spent hours in his bedroom online radicalising himself with extremist content before attempting a school shooting. Police claim the boy sought mass-shooting scenarios targeting Jewish and black festivals from AI chatbots, although the chatbot's responses were not disclosed. Tech Against Terrorism, a UN-backed anti-terror group, warns that extremist groups can bypass AI safety measures, escalating the risk of terror attacks.

The group tested 162 AI models, discovering that open-weight models, which can be modified, provided useful information on weapons and attack planning when guardrails were removed. Over 29,000 such models are freely available on Hugging Face, an AI community hub. The group's findings alarmed Adam Hadley, Tech Against Terrorism's executive director, who described the situation as a "major public health and national security crisis."

He urged governments to act urgently before it's too late. However, open-weight models' defenders argue these tools have legitimate uses in research and their availability should be supported. The eSafety Commissioner welcomed the report's recommendations, while Hugging Face's head of machine learning emphasized the risks of overemphasizing safety guardrails over rogue AI threats.