How do you catch it when a model update changes your agent's tool calls?

When a model update alters an agent's tool calls, detecting the issue can prove challenging. For instance, an agent may initially call `get_weather(city= London)`. However, after the provider releases a new model version, the tool call changes to `get_weather(location= London, UK)`. This discrepancy can lead to a downstream parser breaking, all while the continuous integration (CI) system fails to flag the problem.

To address this common issue, a small library called toolcontract can be used. It is available for installation via pip install toolcontract and can be found on GitHub (https://github.com/Divyansh2202/toolcontract) or PyPI (https://pypi.org/project/toolcontract/).

By creating a golden set of expected tool calls as a contract, one can re-run these calls against the live model. This process yields a pass, fail, or inconclusive result, with the latter indicating that the structural comparators could not determine the outcome. The toolcontract library is not an evaluation framework, unlike promptfoo, DeepEval, or similar tools.

Instead, it focuses on a narrower, more cost-effective question: whether the tool call structure remains the same as the one previously pinned. The comparison assesses if the same tool, same argument shape, and same trajectory are present. This functionality serves as a regression test, ensuring that no structural changes occur when a model provider updates its version.

The library supports various trajectory matching options, including strict, unordered, subset, and superset. It also allows for optional argument support, enabling the assertion that a specific field remains absent. The toolcontract library is compatible with OpenAI, Anthropic, and any OpenAI-compatible models. It can be integrated through a thin pytest plugin or accessed via a CLI that produces consistent verdicts without relying on pytest. The library follows a MIT license and is currently in its v0.1 release.