Judge noise or answer noise? What really moves RAG evaluation scores
Summary. RAG evaluation scores change from run to run even when nothing in the system changes. I measured where that variation comes from in my own system: from the LLM judge grading the same answers differently, or from the generator writing different answers. For faithfulness, about 85% of the run-to-run variance came from the answers. For answer relevancy, nearly all of it did. Context recall…
We haven't written up this one. Dev.to has the full story — the link below goes straight to it.
