{
  "id": 5430796,
  "title": "Eval-First RAG: Use Separate Scores to Triage Failures",
  "url": "https://urgent.news/2026/09/03/eval-first-rag-use-separate-scores-to-triage-failures",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-03T22:20:33.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/nsachok/eval-first-rag-use-separate-scores-to-triage-failures-33ed"
  },
  "original_language": "en",
  "account": "Eval-First RAG: Use Separate Scores to Triage Failures\n\nRelevance and faithfulness are separate diagnostic boundaries in Retrieval-Augmented Generation (RAG). They provide a clearer way to investigate failures rather than relying on a single aggregate number. Retrieval relevance, generation faithfulness, and related metrics help pinpoint the stage where things go wrong, allowing targeted improvements.\n\nBuilding the evaluation harness early in the pipeline enables easier observation of candidates, ranking, context supplied, and final answers. This approach is more practical than trying to reconstruct issues later on. Recall@K, Precision@K, Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain (nDCG) are useful retrieval metrics that reveal different failure shapes. They distinguish between absences, noise, and poor placements.\n\nWhen retrieval is functioning well but generation fails, focus on document representation, search, reranking, or the corpus. This targeted inspection narrows down the problem areas. Faithfulness evaluation assesses the relationship between an answer and the supplied context. A relevant but wrong answer may still be acceptable if it's grounded in the provided text, even if it fails to answer the original question.\n\nThe scorecard inherits decisions about relevance and acceptability from the evaluation process. These decisions impact the quality of the scorecard. The generated corpus uses ground truth labels written alongside the documents and questions, providing provenance for each label. However, this approach is specific to synthetic settings and does not guarantee neutrality or real-world truth.\n\nTask labels can have subtler issues, such as visual tasks appearing answerable from text-only data. The corpus includes a check to identify this problem, but its results are not reported here. The design requirement is to test whether a declared modality is actually needed, not just to assign a label.\n\nLLM judges need calibration to ensure their outputs are useful. Calibration involves adjusting the judge's severity ordering and top findings to align with their intended purpose. A separate LLM can handle grading, mapping the evaluator's scores to pass or fail using structured JSON rubrics. This separation allows for independent calibration of the grading model without affecting the generation model.\n\nHaving a separate grading model from the generation model helps avoid confusion between model changes and policy changes. For high-stakes applications, human review of judge decisions can be combined with automated grading. Including a refusal option in the evaluation is important to reward justified cases where no answer should be provided. This approach ensures a comprehensive evaluation of the RAG system, identifying areas for improvement and preventing false positives in the grading process.",
  "summary": "An end-to-end RAG score can show that answer quality changed. By itself, it cannot tell you where to begin investigating. The evidence returned to the model may be wrong, or the model may have mishandled good evidence. One aggregate number hides that distinction and leaves retrieval, reranking, prompting, and generation competing for attention. I use retrieval relevance and generation…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}