Case Study: A Cassette Proxy That Records a Model Once and Replays It in CI
Your integration tests should not need a network, an API key, or a budget to prove that your prompt-building code still assembles the right payload. The cheapest way to get there is to call the model exactly once, store the response under a hash of the request, and replay that response forever after. This is a case study of that small tool: what it does, the code that implements it, where it…
The integration test suite, once a collection of pure functions, gradually evolved into an HTTP client with a credit card attached. This transformation gave rise to predictable issues: CI failures when the provider experienced downtime, pull requests from forks failing due to withheld secrets, and the eventual addition of @pytest.mark.skip to the solitary test that could have detected a broken payload.
The underlying problem was not a model issue, but rather a boundary problem, one that could be resolved with a straightforward approach.
The objective was to create a tool that could perform four specific tasks without any dependencies: compute a stable cache key from a request body, forward the request upstream in record mode, store the response, and serve it in replay mode. Importantly, missing cassettes should fail loudly rather than silently falling back to the network, as this could reintroduce the very flakiness that was being eliminated.
The resulting tool, implemented as a single file called cassette_proxy.py, is designed to run on Python 3.10+ and interact with any OpenAI-compatible chat endpoint. It buffers responses, so streaming functionality is not included in this version. The tool operates in three modes: record, replay, and off.
In record mode, the tool calls the upstream endpoint, saves the response, and returns it. In replay mode, it reads from disk only; if the response is not found, it returns HTTP 409 with the cache key. In off mode, it passes everything through, allowing live model behavior when desired. The tool's code is deliberately simple, consisting of only standard library imports, one file, and three modes, making it easy to understand and debug even at odd hours.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
