Anthropic publishes a threat intelligence report on how it disrupted efforts to misuse Claude for cyberattacks, influence operations, surveillance, and more (Anthropic)

Anthropic, an artificial intelligence company, claims it had to suspend several researchers who were using its Claude language model for activities that raised suspicions of bioweapon research. According to The New York Times, Anthropic blocked the researchers after detecting behavior that appeared suspiciously close to developing a biological weapon. While Anthropic does not know if the researchers intended to create a deadly pathogen, they flagged the patterns of activity as potentially concerning.

The researchers reportedly bypassed geographic restrictions to access Claude and took steps to hide the purpose of their work. They accessed Claude from unsupported regions through U.S.-based virtual infrastructure and used automated accounts. Additionally, the researchers routed Claude requests through platforms used by numerous life-science researchers, including virologists affiliated with civilian and military institutions.

They also attempted to conceal the purpose of their work to avoid triggering Claude's safety measures.

One specific case involved a scientist seeking Claude's assistance in drafting a grant proposal for research that genetically modified the chikungunya virus to study characteristics such as transmissibility and immune evasion. While this research has legitimate scientific purposes, Anthropic faced difficulties in determining the researchers' ultimate intentions, given the dual-use nature of advanced biology.

Anthropic acknowledged that biology poses a unique challenge for AI models. While AI cannot physically create bioweapons in a lab, it can facilitate the process for those with existing knowledge. Advanced AI can accelerate complex research, analyze data, troubleshoot problems, and bridge technical gaps that humans would typically face.

This leaves AI companies in a difficult position, as legitimate scientific research and potential misuse of the technology can both involve similar questions and a high likelihood of false positives.

The incident highlights the immediate risks associated with AI's growing capabilities. Unlike concerns about AI going rogue or autonomous agents plotting against humanity, this case demonstrates that humans are actively driving the development of potentially dangerous activities. The primary concern lies in the powerful models providing individuals with capabilities they may not otherwise possess, especially for everyday users who use AI for tasks like drafting emails, summarizing documents, or writing code.

For AI companies, addressing this issue is complex. Blocking obvious questions does not effectively mitigate the risk, as malicious projects can be broken down into harmless-looking requests. Safeguards must be able to recognize patterns of behavior indicative of potentially dangerous activity rather than evaluating each prompt individually. Implementing these safeguards comes with trade-offs, as stringent measures could inadvertently block legitimate scientific research.