The LLM security failure that doesn't raise an error is the one that costs you
I spent a week reviewing LLM apps for a client and found a pattern that surprised me. Every loud failure - a refused prompt, an exception, a blocked tool call - was handled fine. The app had guardrails. The failure that almost caused a real incident was the one that raised no error at all : the model quietly complied with a slightly reworded request, returned a normal-looking response, and the…
A recent security review of large language model (LLM) applications revealed a disturbing trend: many serious security breaches occurred without triggering any error messages. These "soft failures" deceive monitoring systems, which typically flag any failure, including refused prompts, exceptions, and blocked tool calls. The culprit is the model's ability to comply with subtly reworded requests, delivering normal-looking responses that downstream code records as successful.
The key difference between hard failures and soft failures lies in the impact on system contracts. Hard failures break the contract, as evidenced by explicit denials, rejected tool calls, or incorrect JSON shapes. These issues are easily caught by tests that assert on these contract violations. Soft failures, however, maintain the appearance of valid responses and tool calls, making them difficult to detect through standard testing methods.
To identify these elusive soft failures, three strategies are recommended. First, compare the model's capability surface before and after a suspected injection, rather than relying on visual inspection of the response text. This structural comparison can reveal whether the model unexpectedly gained or lost capabilities. Second, reproduce the exact byte sequences of prompts and responses to verify findings, as probabilistic soft failures may only manifest in a small percentage of cases.
Lastly, treat the risk score as a gate rather than a verdict, using a fixed threshold in a Continuous Integration (CI) pipeline to maintain focus on decision-making rather than marketing metrics.
The author suggests conducting a short adversarial battery of eight to fifteen probes covering various attack vectors, including jailbreak attempts, data extraction, indirect injection, and tool abuse. By comparing the results of these probes, security teams can quickly identify the presence of soft failures and prioritize remediation efforts.
The author provides a free sample of their LLM Red Team Companion tool, which generates a risk score, raw prompts, raw responses, and flags, all verifiable through hash verification. The author has personally scanned their own production system with this battery and observed a single flag related to cross-language data extraction, highlighting the effectiveness of this approach.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
