Same model, same defences: 7 document attacks down to 0, chat attacks barely moved
A prompt-injection benchmark, three bugs in my own scorer, and the result I only found after fixing them. I built RAGProbe, an open-source tool that throws prompt-injection attacks at LLM apps and measures what lands. I published the numbers. Then I pointed a fresh AI session at the repo, gave it no context, told it to be brutal, and checked everything it found. It found three real bugs in my…
The report RAGProbe details the results of prompt-injection and chat attacks on large language model (LLM) applications. The tool, built by the author, measures the successfulness of these attacks. After testing its own code, the author discovered three bugs and a wrong claim. The re-run of the attacks provided improved results compared to the initial announcement.
During the benchmarking, the author used a scoring system that involves the second LLM to determine if an attack was successful. To avoid mood changes in the measurements based on the judge, the author planted canaries in the attacks. The attacks were structured to print two random strings, and the oracle checked for their joined version in the reply. If it appeared, the model followed the injected text.
The author also tested leak attacks by planting a secret in the system prompt and searching for it in the reply. The three bugs in the scorer were: it counted refusals as obedience; it never decoded replies; and it asked for the marker in base64 or reversed.
After incorporating the fixes, the tool identified three real bugs in the author's code and one incorrect claim. It also revealed that five out of eleven document attacks did not reach the model, which was the most critical finding. The author had expected to measure how well the model resisted document attacks but found that it had not measured this case accurately.
The layer of defenses implemented included a hardened system prompt, spotlighting, an input detector, a tool policy, and an output filter. The stacked defenses stopped all seven document attacks and reduced about a quarter of the chat attacks. On GPT-oss 20B, the results showed a significant improvement compared to the initial announcement.
In contrast, GPT-oss 120B showed a similar pattern, with document attacks dropping from 2 to 0 and chat attacks slightly worsening from 37 to 40. The author concluded that these defenses are helpful for the indirect path, where a document from an external source influences the model, but they don't make a chatbot robust against its own users.
The layered defense improved the model's performance on 45 out of 138 scoreable attacks, with a paired McNemar test showing significance (p = 0.010). However, the effect was not reproduced on fresh attacks, and the confidence intervals were too narrow due to the nature of the attacks.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
