Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing
Semantic query engines ask a large language model (LLM) to judge individual rows, and benchmarks compare these engines on the quality of a few executions at temperature 0. We measure how much another execution changes. We re-execute the text and tabular workloads of SemBench, LRO-Bench, and UDA-Bench through their own harnesses, log every one of 1.2 million model calls, and reanalyze SemBench's…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
