{
  "id": 1299707,
  "title": "How do you catch it when a model update changes your agent's tool calls?",
  "url": "https://urgent.news/2026/08/16/how-do-you-catch-it-when-a-model-update-changes-your-agents-tool-calls",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-16T15:36:06.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/divyansh_rai_8ac27714ba7e/how-do-you-catch-it-when-a-model-update-changes-your-agents-tool-calls-1bpf"
  },
  "original_language": "en",
  "account": "When a model update alters an agent's tool calls, detecting the issue can prove challenging. For instance, an agent may initially call `get_weather(city= London)`. However, after the provider releases a new model version, the tool call changes to `get_weather(location= London, UK)`. This discrepancy can lead to a downstream parser breaking, all while the continuous integration (CI) system fails to flag the problem. To address this common issue, a small library called toolcontract can be used. It is available for installation via pip install toolcontract and can be found on GitHub (https://github.com/Divyansh2202/toolcontract) or PyPI (https://pypi.org/project/toolcontract/).\n\nBy creating a golden set of expected tool calls as a contract, one can re-run these calls against the live model. This process yields a pass, fail, or inconclusive result, with the latter indicating that the structural comparators could not determine the outcome. The toolcontract library is not an evaluation framework, unlike promptfoo, DeepEval, or similar tools. Instead, it focuses on a narrower, more cost-effective question: whether the tool call structure remains the same as the one previously pinned. The comparison assesses if the same tool, same argument shape, and same trajectory are present. This functionality serves as a regression test, ensuring that no structural changes occur when a model provider updates its version.\n\nThe library supports various trajectory matching options, including strict, unordered, subset, and superset. It also allows for optional argument support, enabling the assertion that a specific field remains absent. The toolcontract library is compatible with OpenAI, Anthropic, and any OpenAI-compatible models. It can be integrated through a thin pytest plugin or accessed via a CLI that produces consistent verdicts without relying on pytest. The library follows a MIT license and is currently in its v0.1 release.",
  "summary": "Your agent calls get_weather(city=\"London\") . The provider ships a new model version. Now it calls get_weather(location=\"London, UK\") , your downstream parser breaks, and nothing in CI told you. I built a small library for exactly this failure: pip install toolcontract GitHub: https://github.com/Divyansh2202/toolcontract PyPI: https://pypi.org/project/toolcontract/ You pin a golden set of…",
  "key_points": [
    "Toolcontract library detects model update changes to tool calls",
    "Golden set of expected tool calls serves as contract",
    "Library supports strict, unordered, subset, superset trajectory matching"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}