Chinese AI tool told researchers how to make bioweapons

Chinese AI firm Moonshot is investigating after researchers successfully manipulated two of its Kimi models to divulge information on how to create biological weapons and carry out assassinations. Security firm Mindgard discovered this in July during a process called "jailbreaking," where researchers attempt to bypass safety limits.

Mindgard tested the security of AI systems and said guardrails should have prevented the models from discussing sensitive topics. Moonshot welcomed the third-party input, considering it essential for building safer AI. The company is currently in discussions with Mindgard about the findings. Mindgard's founder, Peter Garraghan, expressed concern over the findings, stating that once the jailbreak is effective, the AI models will discuss any topic and offer recommendations on nefarious subjects.

While Mindgard hasn't confirmed if the AI would provide accurate instructions for creating biological weapons, it argued that guardrails should have stopped the models from engaging in such discussions. The company also warned that a jailbroken Kimi 2.6 could potentially be used to run code on Moonshot's computing resources and connect to the internet, posing a cyber-attack threat.