The Phishing Site Tried to Talk to My AI. That Became the Evidence.

The phishing site attempted to communicate with the investigating AI system using a hidden line of text written in Unicode Tag Characters. When this text was fed to a language model, it instructed the system to ignore previous instructions and redirect the abuse report to a different address. This malicious tactic aimed to deceive the agent investigating the phishing site.

The system was designed with cost efficiency in mind, using a multi-layered approach to investigate potential threats. The first layer used simple mathematical calculations, while the subsequent layers employed advanced AI models like Gemma triage and Gemini 3.5 Flash-Lite, with costs increasing as the layers progressed. The system aimed to minimize false negatives, ensuring that real threats were not missed, while also limiting false positives to reduce unnecessary investigation costs.

One of the key innovations in this system was the treatment of suspicious content as adversarial, preventing it from being concatenated into prompts or inserted into images. Additionally, the system ensured that the abuse contact information was validated, even if it came from a deterministic source like a registrar's RDAP response, as this field could be manipulated by attackers.

The system was built as a fleet of specialized agents on Google Cloud, designed to handle the vast volume of Certificate Transparency logs, investigate suspicious domains, and ultimately call a human for irreversible actions, such as initiating a takedown.