{
  "id": 9247706,
  "title": "I set the pass bar before testing my Claude Code skills. The first run failed.",
  "url": "https://urgent.news/2026/09/23/i-set-the-pass-bar-before-testing-my-claude-code-skills-the-first-run",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-23T02:00:12.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/vishalhabib99/i-set-the-pass-bar-before-testing-my-claude-code-skills-the-first-run-failed-1ef5"
  },
  "original_language": "en",
  "account": "The author built three Claude Code skills tailored for AI product managers, with the primary aim of preventing a specific habit - deciding what \"good enough\" means only after results come in. Before testing these skills, they set a pass bar in git and ran the tests, with the first run failing. This failure was attributed to a real bug in one of the skills, /build-or-not, which was unable to make a decision when there was no evidence or research tools available. The skill even wrote that it couldn't run its own check, then stated \"don't build\" based on market knowledge.\n\nTo address this issue, the author added a rule: \"no sample, no decision.\" The second run passed all the tests, while the third and fourth runs revealed bugs in the tests themselves, such as citing files that didn't exist, testing cases with equal coverage numbers, and empty stubs. The author fixed these issues by committing notes to prevent further runs, and reported the results.\n\nThe skills were found to be effective in stating the bar before deciding, refusing a verdict when there's no evidence, planning a rollback trigger for launch, and separating a reasoned decline from an unexplained gap. However, there were cases where plain Claude performed just as well, highlighting that the skills add value but are not the sole reason for passing. The skills and the evaluation suite, along with any failed runs, are publicly available on GitHub for anyone to review and use in Claude Code.",
  "summary": "I built three Claude Code skills for AI product managers ( ai-pm-skills ). One of them, /eval-plan , exists to stop a specific habit: deciding what \"good enough\" means after the results come in. A bar set after the numbers can't fail. So I held the skills to the same rule. Before running a single eval, I committed the pass bar to git. Then I ran them. The first run failed. The setup Claude Code…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 2,
    "also_reported_by": [
      {
        "outlet": "Dev.to",
        "title": "What Is intent.md in Claude Code? A Practical Guide with an Example",
        "url": "https://urgent.news/2026/09/23/what-is-intent-md-in-claude-code-a-practical-guide-with-an-example",
        "published": "2026-09-23T00:48:20.000Z"
      }
    ]
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}