When AI models refuse to fight hackers

As cybersecurity teams scramble to combat AI-driven hacking threats, they face a perplexing challenge: the very AI models they depend on for investigation can inexplicably refuse to assist. While AI leaders rush to develop rapid-response tools for hacking and other online malfeasance, strict safeguards on their most potent models abruptly impede defenders during critical investigation phases.

OpenCode, an open-source AI coding agent, fell victim to a large-scale distillation attack over the weekend, enabling attackers to obtain API keys from some customers, including access to OpenAI, Anthropic, and Google services. OpenCode co-founder Dax Raad reported that attackers had been targeting its systems for 48 hours straight, compelling the company to continuously run AI agents to counteract the breach.

OpenAI, Google, and Anthropic stated they collaborated with affected customers to invalidate exposed API keys and rectify unauthorized usage charges. Raad noted on X that the frontier AI models initially aided the breach investigation but subsequently began obstructing requests as the probe shifted toward potentially malicious code, even though OpenCode sought to defend its own systems.

OpenCode subsequently gained entry to OpenAI's Daybreak Blue program, granting vetted cybersecurity defenders access to models with reduced restrictions. Raad emphasized the necessity for more proactive measures, highlighting significant improvements implemented in recent days. The incident highlighted a conundrum for cybersecurity teams weighing the use of powerful proprietary AI models, which come with built-in limitations, against open-weight alternatives offering greater control over deployment.

When Hugging Face detected suspicious activity on its systems, later traced to OpenAI's own agents, its security team turned to a Chinese open-weight model for investigation assistance after encountering restrictions on frontier models. AI companies argue they must restrict access to their most potent models to prevent malicious actors from exploiting their cyber capabilities.

Cybersecurity teams must navigate the balance between the need for swift AI-powered defenses and restrictions designed to keep those same capabilities out of attackers' hands, according to Chris Jones, chief technology officer at Nightwing. "We're all going to have to figure out how that race gets modulated over time," Jones stated.