{
  "id": 2048683,
  "title": "A Good LLM Exam Is 90% Traps",
  "url": "https://urgent.news/2026/08/20/a-good-llm-exam-is-90-traps",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-20T00:04:15.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/ramses203/a-good-llm-exam-is-90-traps-4faj"
  },
  "original_language": "en",
  "account": "Last time, the author lost five times when giving their LLM an order-reading exam. Today, they explain how the exam was built. The conclusion is that nice questions are a waste of paper; instead, focus on the happy path. Start with a question like \"5 boxes of the 250 shipping boxes please → shipping box 250, 5 boxes.\" The exam should aim to trap the model in its worst accidents, such as shipping something nobody ordered. The most important trap is after the model has learned. When a human picks a match once, the model remembers it, leading to accidents. To build the exam, write down the worst accidents first and create questions to cause them. Plant near-twins in the data and include after-learning questions that reveal learning scenarios. Normal cases come last, as they should pass anyway. If the exam is nice, it will pass in production, allowing accidents to occur.",
  "summary": "Last time I gave my LLM an order-reading exam and lost 5 times as the exam author. Today: how that exam was built. Conclusion first — nice questions are a waste of paper. You'll want to start with the happy path Ask anyone to write a test and they start with the case that works. \"5 boxes of the 250 shipping boxes please\" → shipping box 250, 5 boxes. It passes. Feels good. Reassuring. But that's…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}