{
  "id": 15369483,
  "title": "Pin the Contract, Not the Model: A Bootcamp Lab on Provider-Swappable Agent Runs",
  "url": "https://urgent.news/2026/10/18/pin-the-contract-not-the-model-a-bootcamp-lab-on-provider-swappable",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-18T07:08:12.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/hackjs_7468/pin-the-contract-not-the-model-a-bootcamp-lab-on-provider-swappable-agent-runs-4dga"
  },
  "original_language": "en",
  "account": "This lab teaches an essential principle for grading student agent runs: the rubric should focus on the contract between the agent and the model, not on the model itself. The key takeaway is to record the agent's interactions with the model once, replay them forever, and grade the results offline.\n\nMany students encounter the same issue early in the bootcamp: they build an agent loop against whatever free model is available that day, submit their work, and then expect the grader to produce consistent results. However, when the grader re-runs the submission, it often yields different answers, tool calls, timeouts, or rate-limit pages. The rubric, in this case, is measuring the luck of the re-run, not the student's design.\n\nThe solution lies in pinning the contract, not the model. This means that prompts are hashed, so the same prompt string, regardless of whitespace, maps to the same key. Responses are recorded to a plain JSON cassette, which is committed to the repository. During grading, the process runs on replay only, with no network calls, API keys, or clock dependencies. One live run at the end is allowed as a demonstration, but it should never be the sole source of truth.\n\nTo implement this, the lab requires four files containing roughly fifteen minutes of code. The \"contract\" is defined as a Protocol, which can be any object with a complete method. The lab/providers.py file outlines the structure for the provider, which includes a prompt_key function to normalize whitespace, a Provider protocol with a complete method, and a CassetteMiss exception for when there is no recording for a prompt. The replay provider checks the cassette before calling out, preventing wasted calls when re-running a half-finished session.\n\nThe Recorder class wraps any live provider and appends what it sees, ensuring that re-running a session with existing recordings doesn't burn calls. The live adapter is a sketch that needs adaptation before use, including endpoints, auth headers, and response shapes, which vary per provider. By following these four rules, the class can stop fighting flaky re-runs and focus on grading the student's design rather than the performance of a specific model.",
  "summary": "A lab that grades an agent run should never care which model produced that run. If your rubric silently changes when the free tier changes, you were grading the vendor, not the student. That is the whole thesis of this lab, and it fits in one sentence: record once, replay forever, grade offline. The failure I keep watching students repeat Every bootcamp cohort hits the same wall around week two.…",
  "key_points": [
    "Focus rubric on agent-model contract, not model itself",
    "Pin prompts with hashing to ensure consistent grading",
    "Implement recording and replay system to eliminate flaky re-runs"
  ],
  "editors_take": "Decoupling agent evaluation from specific model performance ensures that grading focuses on the student's architectural design rather than the unpredictable fluctuations of live API responses.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}