Someone used Claude to build a potential bioweapon. The real threat is much deeper

Anthropic, an AI company, recently revealed that scientists attempted to use their flagship AI model, Claude, to research potentially deadly bioweapons. While there is no evidence that the scientists intended to cause harm, the possibility of such a scenario growing more likely as AI models become more powerful and advanced is concerning.

The report from Anthropic highlights various instances of scientists attempting to use Claude for research that could have serious consequences. For example, scientists reportedly sought help for a "gain of function" research project on the mosquito-borne Chikungunya virus, which involves making a pathogen more deadly or easier to spread.

While this type of research can have legitimate scientific purposes, asking for help with something nefarious in the guise of a request for something else is a form of "jailbreaking" – a method of tricking an LLM into providing information or guidance it should not. The report also mentions instances of scientists trying to develop "novel venoms and toxins" using Claude.

Dual-use technology, such as developing new toxins for legitimate purposes like Botox, can also be used for dangerous purposes like creating more powerful tools for covert assassinations. Although Anthropic successfully blocked some potentially harmful requests, it is difficult to know how many similar requests slipped through their filters or guardrails.

The report highlights the challenges of policing LLMs, as it is easy to block specific requests like "make me a stronger version of COVID," but harder to detect when legitimate scientific inquiries are masking attempts to create deadly toxins or new pathogens. As frontier models continue to improve their scientific reasoning abilities, the threat of AI being used to develop powerful new pathogens will only increase.