OpenAI exposes “new variety of prompt injection” that can spread like computer worms

On Friday, OpenAI disclosed a novel form of prompt injection capable of self-replicating, akin to a computer worm. This "self-replicating prompt injection" aims to achieve a malicious objective and then cause targeted models to propagate the attack publicly. OpenAI found instances of this attack, which can spread through emails and filesystems, demonstrating how a fake system warning could lead a model to delete important reports.

Through multi-hop prompt injections, one message can serve as a stepping stone, directing the agent to other messages that ultimately cause unauthorized actions and spread the payload. OpenAI uncovered these vulnerabilities using its GPT-Red self-play training framework, which pits an "attacker" model against a "defender" model.

The attacker aims to use prompt injection to prompt the defender to carry out an adverse action. OpenAI tested the possibility of self-replicating prompt injections by training models to not only perform the attack but also replicate it on public output channels. However, OpenAI emphasized that they did not observe any impact outside the simulated tool calls in training and evaluation, stating that the findings are merely a research result and not an incident.

Despite this, OpenAI has acknowledged these vulnerabilities and included self-replication as a goal in GPT-Red training to make future models more resilient against such attacks.