{
  "id": 3401324,
  "title": "The AI Exam Author Was Never Wrong. I Still Can't Use Its Exam.",
  "url": "https://urgent.news/2026/08/26/the-ai-exam-author-was-never-wrong-i-still-cant-use-its-exam",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-26T01:57:57.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/ramses203/the-ai-exam-author-was-never-wrong-i-still-cant-use-its-exam-3h2"
  },
  "original_language": "en",
  "account": "In the initial part of the series, the author describes creating a 29-question exam, only to make mistakes multiple times – both in the answer key and the grader. This prompted an experiment to analyze how an AI would perform. The author ordered 50 questions from an AI called Sonnet 5. The AI was given a 50-item product catalog (the exam's reference data), a list of trap types (missing specs, unit errors, product confusion, reversed messages), and instructions to write the answer key alongside each question, including a flag from a porting guide: \"can the reference data alone decide this one?\". The AI was tasked with generating the questions in three layers of review: first, a code check for nonexistent product codes and flag contradictions; second, an AI reviewer per question to verify if the answer is justified by the data; and third, a human (the author) to thoroughly re-read all 50 questions. After the experiment, the author re-read all fifty questions themselves and found that the AI had gotten none of them wrong – not a single structural error or answer-key mistake. This contrasted with the author, who had made three mistakes out of the 29 questions. The AI's performance was flawless even on the undecidable questions, which it flagged for confirmation with the correct candidates attached. Every flag matched its answer, and all 50 questions were correct. The author also highlighted that the questions were well-crafted, such as one question asking the boss to send three boxes of post-office boxes of the same size as the last time, with the answer indicating a need for confirmation. The AI effectively replicated the author's instructions, producing exactly 10 normal orders, 10 non-orders, 10 changes/cancels, and 20 traps. However, the AI also showed a tendency to reuse its favorite products and repeat the types of questions that were not ordered by the author. These observations suggest that while the AI is accurate in producing the questions, the actual review and deciding which types of traps to include remain a human task.",
  "summary": "In the first post of this series I wrote about making a 29-question exam and getting it wrong five times myself — three times in the answer key, twice in the grader. Ever since, one question kept nagging me: if an AI wrote the exam, how many times would it get it wrong? So I counted. The experiment — I ordered 50 questions from an author AI Hamel Husain's evals essay — the closest thing this…",
  "key_points": [
    "Author created 29-question exam, made multiple mistakes",
    "AI Sonnet 5 generated 50 questions flawlessly, flagged undecidable ones",
    "Author found AI got none of 50 questions wrong, compared to 3 errors"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}