Microsoft built a prompt injection detector. Then it caught a phishing campaign instead.

Microsoft has developed a system to identify a type of prompt injection attack, but an earlier phishing campaign revealed the potential dangers of this technique. Attackers have been inserting invisible Unicode tag characters into email bodies, which render differently than intended text, allowing them to bypass spam filters and AI-based text classifiers.

These characters can split keywords and change how software processes the text. Microsoft's research team documented a similar tactic called "ASCII Smuggling," which targets large language models (LLMs) by using Unicode tag characters to alter the underlying string that software processes. In the phishing campaign detected by Microsoft Defender for Office 365, the attackers used these characters in high-signal financial terms such as "funding," "loan," and "credit" to evade filters.

The system flagged around 21,000 messages the day before the campaign began, and the number of flagged messages increased to over 1.3 million the following day and 2.3 million just two days after that. The campaign demonstrates how such attacks can slip through unnoticed, as recipients see seemingly ordinary offers for business loans and credit lines.

When natural language processing systems tokenize text, they break it into smaller units, and the insertion of unexpected Unicode characters can alter these tokens. Different tokenizers may handle these characters differently, requiring developers to test their specific models. Normalizing the text through Unicode normalization techniques like NFC and NFD may not always remove the tags, which means they can still pass through to the next stage of processing.

Email providers typically have other methods to detect suspicious messages beyond just the content, but AI pipelines may rely more heavily on the text itself. This can pose a problem when AI agents process outside text and make decisions based on it, as invisible characters can change how the text is processed. To mitigate this risk, developers should remove characters in the specified Unicode range before the text reaches the model, but this can be more challenging if the application legitimately requires those characters.

Developers should also compare the original text with a version stripped of the tags and test their tokenizer to understand how it handles these characters.