Pin the Contract, Not the Model: A Bootcamp Lab on Provider-Swappable Agent Runs
A lab that grades an agent run should never care which model produced that run. If your rubric silently changes when the free tier changes, you were grading the vendor, not the student. That is the whole thesis of this lab, and it fits in one sentence: record once, replay forever, grade offline. The failure I keep watching students repeat Every bootcamp cohort hits the same wall around week two.…
This lab teaches an essential principle for grading student agent runs: the rubric should focus on the contract between the agent and the model, not on the model itself. The key takeaway is to record the agent's interactions with the model once, replay them forever, and grade the results offline.
Many students encounter the same issue early in the bootcamp: they build an agent loop against whatever free model is available that day, submit their work, and then expect the grader to produce consistent results. However, when the grader re-runs the submission, it often yields different answers, tool calls, timeouts, or rate-limit pages. The rubric, in this case, is measuring the luck of the re-run, not the student's design.
The solution lies in pinning the contract, not the model. This means that prompts are hashed, so the same prompt string, regardless of whitespace, maps to the same key. Responses are recorded to a plain JSON cassette, which is committed to the repository. During grading, the process runs on replay only, with no network calls, API keys, or clock dependencies. One live run at the end is allowed as a demonstration, but it should never be the sole source of truth.
To implement this, the lab requires four files containing roughly fifteen minutes of code. The "contract" is defined as a Protocol, which can be any object with a complete method. The lab/providers.py file outlines the structure for the provider, which includes a prompt_key function to normalize whitespace, a Provider protocol with a complete method, and a CassetteMiss exception for when there is no recording for a prompt.
The replay provider checks the cassette before calling out, preventing wasted calls when re-running a half-finished session.
The Recorder class wraps any live provider and appends what it sees, ensuring that re-running a session with existing recordings doesn't burn calls. The live adapter is a sketch that needs adaptation before use, including endpoints, auth headers, and response shapes, which vary per provider. By following these four rules, the class can stop fighting flaky re-runs and focus on grading the student's design rather than the performance of a specific model.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
