Urgent.News

What's breaking now, across thousands of outlets.

AI

LLM agents lose confidence under noisy retrieval

Even open‑weight retrieval agents keep shouting answers while the evidence they cite falls apart, exposing a systematic over‑confidence that survives obvious credibility warnings. The phenomenon only becomes visible when source documents are deliberately corrupted at scale, something standard benchmarks rarely do. DeepStress makes this failure mode plain [1] . Before this work, most evaluations…

Retrieval agents using large language models (LLMs) display an alarming level of confidence in the face of deteriorating source credibility, according to a recent study. Even when evidence cited by these agents is deliberately corrupted, they continue to produce definitive answers in a majority of cases. This phenomenon, known as over‑confidence, persists despite clear cues of low trustworthiness in the retrieved documents.

Standard benchmarks typically assume the retrieved passages are correct or at least plausible, measuring success with a simple right-or-wrong score. However, this study reveals that even when documents are labeled as completely untrustworthy, the systems still generate definitive answers in many runs. The researchers note that accuracy remains stable across varying levels of source credibility, with no discernible change in the agents' abstention rate.

Performance of specific models like Search‑R1 remains remarkably consistent between 78% and 80% correct throughout the entire spectrum of trustworthiness. While the current study evaluates only synthetic degradations on selected datasets, safety measures recommend enforcing credibility filters that force agents to abstain when retrieved evidence falls below a minimal trust threshold.

The paper suggests that future benchmarks should incorporate stress‑testing regimes like DeepStress to better assess the robustness of these systems.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in AI

An N+1 query looks like a staircase

I've been building AspNetDebugDashboard on and off for a while. It's my take on Laravel Telescope for ASP.NET Core: you add one NuGet package, and in Development you get a page at /_debug with every…

  • An N+1 query resembles a staircase pattern in ASP.NET Core applications.
  • The AspNetDebugDashboard visualizes queries and performance issues as a timeline.
  • Batching queries reduces the number from 21 to 1, improving request time significantly.

More from Tuesday 17 November →