Synthetic vs Real Test Cases for LLM Agent Evals
Originally published on robatdasorvi.com Synthetic test cases are good enough to rank LLM agents against each other. They are not good enough to certify them. Real production traces catch failures that generated cases almost never contain, so the final release gate should run on real cases. On 2026-09-22, X suspended my approval-gated X growth agent's account a second time, one week after the…
Synthetic test cases are sufficient for comparing the relative performance of language model agents. However, they are insufficient for establishing rigorous certification. Real production traces uncover failures that synthetic cases rarely encompass, suggesting that a final release gate should utilize genuine scenarios. On 2026-09-22, X suspended the author's approval-gated X growth agent account for a second time, one week after tightening the reply cap from 16 to 150.
The lockout check failed to trigger despite matching none of its known selectors, as the real suspension page did not align with them. The selector list was manually curated, adept at covering the cases envisioned by its creators but failing to account for unexpected inputs. A synthetic evaluation suite suffers from the same vulnerability.
Research indicates that synthetic versions are valid but less challenging for LLMs than human-written originals (arXiv 2505.22830). Synthetic test collections prove effective for relative ranking among LLM agents but less reliable for absolute scores (Rahmani et al., 2025). The bias tends to favor LLM-based retrieval systems when judged using LLM-generated queries and judgments, impacting absolute performance numbers more than relative comparisons (arXiv 2506.10301).
Although this methodology may accurately assess whether agent v2 outperforms v1 on the same suite, it does not accurately gauge whether this agent can complete 95% of real tasks. Synthetic suites artificially elevate scores and are thus widely trusted despite their limitations.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.