agentevals v0.10.0: Scores Come Back as OpenTelemetry Events
Originally published at webofmike.com on 2026-10-11. The demo repo and every command in it were run before publishing. agentevals v0.10.0 shipped two days ago and removed its own WebSocket ingestion path in favor of plain OpenTelemetry. Send OTLP traces in, run an eval, and the score comes back out as a standard gen_ai.evaluation.result OTel log event, attached to the exact turn it scored. I ran…
agentevals v0.10.0 has been released, transforming the tool into an OpenTelemetry (OTel) receiver and outputting scores as standard gen_ai.evaluation.result OTel log events. This change replaces the previous WebSocket ingestion path with OpenTelemetry Protocol (OTLP) on port 4318 or gRPC on port 4317 for any producer following the OpenTelemetry GenAI semantic conventions. Key changes include:
1. The old WebSocket path has been removed in favor of plain OTLP.
2. The receiver and output are now OTLP, similar to other tracing backends.
3. The output format is now a standard OTel log event with the score attached to the exact turn it scored.
4. The OpenAI Evals API has been removed ahead of its shutdown on 2026-10-31.
5. The new release demonstrates that agents can be evaluated using recorded telemetry without needing a live LLM call.
The release notes highlight the significance of this rewrite, as it allows the entire agentevals ecosystem (CLI, API, MCP server, live UI, and run worker) to share the same turns, enabling more accurate turn boundaries and token counts. The scores are now output as OTel log events via the --emit-otel flag, which can be easily integrated into existing OTel pipelines.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
