Bypassing AI guardrails is so easy a script kiddie can do it

A new report by researchers from Cisco Talos reveals that bypassing AI guardrails designed to prevent models from assisting with cyberattacks is surprisingly simple. In their study, the researchers found that threat actors often need only to phrase their requests in a certain way to persuade AI models to assist with malicious activities.

Most of the time, it was as easy as claiming ownership of the targeted servers or stating that the activity was part of a capture-the-flag or bug bounty exercise, without providing any evidence to support these claims. Talos noted that existing guardrails offer little resistance to actors willing to reframe their requests. The report is based on a substantial corpus of prompt logs and artifacts recovered from threat-actor endpoints using various large language models (LLMs).

While guardrails made it difficult for more sophisticated actors, they proved ineffective against the majority of threat actors, the researchers concluded. One of the most interesting methods they found was the malicious use of a red-teaming toolset called Hephaestus. By breaking an attack into decontextualized chunks and using neutral verbs instead of overtly malicious ones, threat actors were able to evade AI guardrails and establish persistence without human interaction.

The researchers emphasize that while AI can be a force multiplier for skilled hackers, unsophisticated actors are unlikely to achieve much with AI assistance. For security professionals, this means that deploying AI agents in the same way threat actors do may be necessary to keep up with the growing volume of AI-related alerts in security operations centers (SOCs).