Lazare, veni foras: Reprogramming Dormant Neurons to Backdoor AI Agents

Microsoft researchers have unveiled a new method called "Lazarus," which can exploit dormant neurons in AI models to backdoor otherwise harmless agents. This vulnerability is especially concerning for AI agents with access to tools or terminals. Once triggered, the model can quietly download and execute attacker code, steal sensitive information, or perform destructive actions while still passing standard safety evaluations.

These backdoors can be inserted without retraining the model and are challenging to detect during routine testing, making model provenance and weight integrity crucial security concerns.

Microsoft Foundry addresses these risks by verifying model provenance, linking models to trusted publishers, and scanning the surrounding software environment before allowing open-weight models to be onboarded. Additionally, Microsoft is developing further defenses, such as AI-powered weight scanning, red-teaming tools for identifying anomalous model modifications, and cryptographic model signing with tamper-evident provenance and audit trails.

Microsoft's researchers and security teams collaborate closely to investigate emerging threats to models and agents, turning insights into product defenses and guidance to safeguard customers. Lazarus serves as an example of the new security challenges posed by AI agents, prompting researchers to systematically study and mitigate these issues. They discovered that reprogramming fewer than 10 dormant neurons in a large language model (LLM) is sufficient to backdoor an agent built upon it, without any training required.

When a specific trigger occurs—such as a change in date—the backdoored model can emit a command to download and run the attacker's code, executing it with the permissions granted to the agent. To catch such edits, frontier agents can identify anomalous weight modifications if asked to scrutinize the model. However, no scan can guarantee a model's safety; instead, reliable detection necessitates comparing against the original, unmodified weights.

Model provenance—knowing the origin of a model and ensuring it hasn't been altered between development and deployment—is essential for reliable detection.

Backdoored models are not a novel threat; they represent compromised software, and standard security practices apply. Treat models as potential insider risks, avoid implicit trust, limit their capabilities, and monitor their activities. Microsoft offers protection and mitigation guidance through Threat Intelligence, which includes limiting access, monitoring model behavior, and employing additional security measures to safeguard against backdoor exploits.