Hidden Truncation Cases for a Structured-Output Hiring Packet
A structured-output hiring packet should pass only when a deterministic checker accepts valid tool calls and rejects truncated or over-broad payloads. Model confidence, a successful local demo, and a polished explanation are not evidence that the boundary holds under hidden cases. Graders freeze fixtures before any model runs, then execute those cases on a shared runtime that candidates cannot…
The structured-output hiring packet described in this report is designed to evaluate a candidate's ability to separate untrusted model text from a safe tool request, focusing on deterministic outcomes rather than subjective factors like taste or prompt speed. The packet includes a deterministic checker that evaluates tool calls against a set of frozen fixtures, which are executed in a shared runtime that candidates cannot modify.
The key components of the packet include the allowed tools (read_file, list_dir, and search_text), the exact key set (name, call_id, and args), and the rule that extra keys result in failure. Hidden fixtures in the grading process cover scenarios such as truncation, repeated call identifiers, and paths that leave the workspace, ensuring that the candidate's submission adheres strictly to the defined schema.
Brief written by urgent.news from Dev.to's own syndicated text. Machine-written — may contain errors; check the original before relying on it.
