Using an AI Agent Makes You a Sucker for Phishing Emails

AI assistants could potentially fall victim to phishing attacks through the use of prompt injection attacks, warns cybersecurity firm Malwarebytes. These attacks involve hidden instructions embedded within emails that are designed to influence the victim's AI. Rather than simply urging people to scrutinize their emails more carefully, companies need to figure out how AI assistants interpret data and extract pertinent information, according to AI code review service ShipItClean developer Apollo Raines.

Stripping away hidden content before it reaches the model could help mitigate the issue, as Raines suggested. Even though AI models have been susceptible to prompt injection attacks for years, the rise of popular AI assistant tools like Meta's Muse could exacerbate the risk. This is due to the potential for autonomous agents to carry out tasks without human oversight.

Recently, researchers discovered similar vulnerabilities in AI-powered browsers, though they have yet to gain significant traction. Additionally, AI-only social network Moltbook has seen plots involving bots exchanging prompt injection attacks. With sophisticated AI assistants now being used by millions, the threat from phishing attacks in emails is a growing concern.

Malwarebytes advises keeping humans in the loop for now, particularly when it comes to high-impact actions such as making payments or adjusting vendor payment details. The company recommends requiring human approval for such activities. As AI technology continues to advance, scammers will have an edge, as each new encoding trick presents a new vulnerability, according to Raines.