{
  "id": 19484762,
  "title": "A word-overlap check beat a DeBERTa NLI model at spotting RAG hallucinations",
  "url": "https://urgent.news/2026/11/11/a-word-overlap-check-beat-a-deberta-nli-model-at-spotting-rag",
  "topic": "culture",
  "section": "Culture",
  "published": "2026-11-11T11:01:29.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/guillermo_llopis_591d99e6/a-word-overlap-check-beat-a-deberta-nli-model-at-spotting-rag-hallucinations-39il"
  },
  "original_language": "en",
  "account": "I built three detectors to identify hallucinations in Retrieval-Augmented Generation (RAG) outputs. The lexical detector scores the share of content tokens in a response that do not appear in the source. The numeric detector measures the share of numbers in a response that are not found in the source. The NLI detector uses a cross-encoder model called DeBERTa V3 small to compare response sentences with the closest source sentences. The lexical detector proved to be the fastest and most effective, catching most hallucinations with a high precision of 0.476. The ensemble model, which combined the three detectors, did not outperform the lexical detector alone. The lexical detector's strength lies in identifying plausible-sounding details or new vocabulary not present in the source. The NLI detector struggles with data-to-text tasks, where the source is flattened JSON, and its performance varies depending on the LLM used. The detectors can be used as a triage signal to prioritize human review of flagged responses, rather than blocking or rewriting every flagged answer, which could degrade good ones.",
  "summary": "I expected the neural entailment model to win. It didn't. I wanted to know how far the cheap, no-API-call hallucination checks get on real RAG output before you need an LLM judge or a human. So I built three small detectors, ran them on RAGTruth , and compared them. The one that costs microseconds per response beat the one that costs about seven seconds on a CPU. The setup RAGTruth (Niu et al.,…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}