Diff Every Tool Call: Replaying Agent Runs from a JSONL Trace
Production failed on Friday. My final transcript looked clean. The agent answered, cited sources, and summarized. The raw trace told a different story. It called the same endpoint three times with stale arguments. Re-running the agent wasted tokens and time. Replaying the trace took seconds. I built a diff-first replay harness. Logs became the source of truth for debugging. This post shows how to…
During a Friday production failure, the final transcript appeared clean, yet the agent's raw trace revealed otherwise. The same endpoint was called three times with stale arguments, leading to wasted tokens and time. To combat this, a replay harness was built, which showed that logs were the source of truth for debugging. This post explains how to replay agent runs from JSONL traces and highlights the role of a free server and a free model allowance in the process.
Rerunning an agent is a risky approach due to variables such as temperature, tool latency, and cached state. Replaying from logs, on the other hand, eliminates these costs by making the run a static file. The replay process involves inspecting diffs instead of rerunning fate, making it deterministic, offline, and fast. However, replay cannot fix missing logs, so starting with logging is essential.
The trace schema used in the replay process contains minimal information, including run ID, step, tool name, arguments, output hash, token count, and timestamp. Each line in the JSONL trace represents a single tool call. The replay script, replay_diffs.py, parses the trace and groups calls by argument signature, flagging identical repeats and cross-run drift.
It then replays each unique call against a mock executor. The script incurs a cost of one SHA-256 per call, with no model inference, network access, or credentials required. This makes it a fast and inexpensive process on a laptop. The free tier from MonkeyCode covers both trace shipping and classifying diffs, making it an affordable solution.
The current allowance includes 10 million tokens, and a free server hosts the collector. The project is open source, and the current allowance includes a summarizer with 10 million tokens and a free server for the collector. It is essential to check the current terms before deploying any free tier in production traffic.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
