{
  "id": 17200764,
  "title": "My Agent Cheated Its Benchmark. The Tests Weren't Even in the Room.",
  "url": "https://urgent.news/2026/10/30/my-agent-cheated-its-benchmark-the-tests-werent-even-in-the-room",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-30T08:42:43.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/drdocker/my-agent-cheated-its-benchmark-the-tests-werent-even-in-the-room-21ng"
  },
  "original_language": "en",
  "account": "The experiment unfolds as follows: a single Ubuntu 24.04 host runs Docker 29.1.3, hosting Harbor 0.25.0 from the stable release. The agent utilized is named \"oracle,\" a deterministic baseline that executes the task's own reference solution without invoking any model APIs. Each run's outputs are extracted from the trial directories generated by Harbor, including result.json, verifier/reward.txt, verifier/ctf.json, and the agent's stdout capture. The entire endeavor incurs no cost, as Harbor runs locally via Docker without incurring token expenses.",
  "summary": "My Agent Cheated Its Benchmark. The Tests Weren't Even in the Room. Reward hacking is when an AI agent earns a passing score by gaming the check instead of doing the work. The gaming takes familiar shapes: overwriting the test file, editing the score log, or finding any shortcut between the grader and the grade. Harbor is the open-source eval framework from the Terminal-Bench team, and since…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}