"Trace Shape Reveals Model Capability: 3B vs 7B"

Last week, a reporter traced the same agent using two different model sizes: Qwen2.5-3B with 3 billion parameters and Qwen2.5-7B with 7 billion parameters. The only variables were the model and the task. The reporter did not read any outputs but instead focused on the trace shapes.

For the 3B model, each trace consisted of exactly two spans: one agent.run and one llm.call, with no tool calls. In contrast, the 7B model produced an average of 5.8 spans per trace, involving multiple llm.call and tool.call spans, indicating the use of tools. The 3B model was unable to perform tool calls reliably, attempting to answer directly instead. This resulted in a degenerate trace shape with a straight line from start to finish, lacking branching or tool interaction.

On the other hand, the 7B model exhibited a rich trace shape, exploring multiple tool calls, reading results, and synthesizing information. The trace shape showed branching and iterative behavior typical of an agentic model. The key takeaway is that model traces can reveal a model's capabilities by analyzing the shape, without needing to read the outputs.

A 3B model does not engage in the agentic loop, while a 7B model does. When evaluating models for agentic tasks, it is crucial to consider the trace shape, such as the use of tools, number of steps, and changes in shape with task complexity. This experiment demonstrates that model size significantly impacts agentic capability, with larger models exhibiting richer trace shapes indicative of more sophisticated agent behavior.

This insight is part of the broader thesis of AgentPulse, which posits that trace shape serves as a behavioral fingerprint for detecting capability gaps, model regressions, and other issues, all through metadata analysis without reading prompts or outputs.