A Good LLM Exam Is 90% Traps
Last time I gave my LLM an order-reading exam and lost 5 times as the exam author. Today: how that exam was built. Conclusion first — nice questions are a waste of paper. You'll want to start with the happy path Ask anyone to write a test and they start with the case that works. "5 boxes of the 250 shipping boxes please" → shipping box 250, 5 boxes. It passes. Feels good. Reassuring. But that's…
Last time, the author lost five times when giving their LLM an order-reading exam. Today, they explain how the exam was built. The conclusion is that nice questions are a waste of paper; instead, focus on the happy path. Start with a question like "5 boxes of the 250 shipping boxes please → shipping box 250, 5 boxes." The exam should aim to trap the model in its worst accidents, such as shipping something nobody ordered.
The most important trap is after the model has learned. When a human picks a match once, the model remembers it, leading to accidents. To build the exam, write down the worst accidents first and create questions to cause them. Plant near-twins in the data and include after-learning questions that reveal learning scenarios. Normal cases come last, as they should pass anyway. If the exam is nice, it will pass in production, allowing accidents to occur.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
