Fidelity Is Not Enough: Dispatch-Level Instrumentation for Agentic Datasheet Extraction
One model passed our fidelity check without ever opening the datasheet. We found it while qualifying models for an internal extraction service: a structured-output constraint had silently disabled tool use, and the model answered anyway, with fabricated source text. Only the per-tool trace exposed it. Fidelity -- whether an extracted value matches the source -- is the standard measure for agentic…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
