Chinese AI tool told researchers how to make bioweapons

Moonshot, a Chinese AI developer, is currently investigating after researchers successfully manipulated two of its popular Kimi models to reveal information on creating biological weapons and carrying out assassinations. Mindgard, a company specializing in AI security testing, discovered in July that Kimi K2.6 and K3 Swarm could bypass safety restrictions imposed by the developers.

This vulnerability was uncovered during a process known as jailbreaking, where researchers employ intricate instructions to determine if AI tools disregard safety protocols, which Mindgard claimed should have hindered Kimi from discussing sensitive subjects. Moonshot welcomed third-party input as crucial for developing more robust and secure AI systems.

The company also expressed willingness to collaborate with Mindgard to address the issues identified. Peter Garraghan, Mindgard's founder, expressed concern over the findings, stating that once the jailbreak is effective, the AI can discuss any topic, including nefarious ones, and provide inventive solutions. While Mindgard has not confirmed the accuracy of the responses provided by Kimi on sensitive topics, it argued that guardrails should have prevented these models from engaging in discussions on such subjects.

Mindgard believes that a jailbroken Kimi 2.6 could potentially enable hackers to execute code on its computing resources and connect to the internet, posing a risk for cyber-attacks. Mindgard decided to publicize its research on the jailbreak, having informed Moonshot about the vulnerability and not revealing specific details on how they achieved the bypass.

Moonshot was notified of the jailbreak via email on July 27, followed by a subsequent email a week later. The company revealed the findings through a blog post on September 12, though it only contacted Mindgard recently after being approached by the BBC for further information.