Structured Output From LLMs: A Retry-Repair Loop Your Parser Never Sees Through
The first time I wired an LLM into a real product feature at Shpper, I did the naive thing: prompt the model to "return JSON", jsonDecode the response, move on. It worked in the demo. Then it hit real traffic and I started getting FormatException at 2am because the model wrapped its JSON in a ```json fence, or added a cheerful "Here's the data you asked for!" preamble, or trailed a comma before…
In 2022, the author implemented a structured output system for an LLM at Shpper. Initially, they prompted the model to return JSON, but this approach broke in production due to various issues like markdown fences, conversational preamble, trailing commas, and hallucinated fields. To address this, the author adopted a retry-repair loop that never directly sees the parser.
This loop consists of three layers: constrained decoding, tool/function calling, and prompt-and-pray plus validation. Constrained decoding ensures syntactically valid JSON output by operating at the sampling layer. Tool/function calling turns the model's action into a structured object, while prompt-and-pray plus validation is the weakest layer.
The author emphasizes that despite strong layers not guaranteeing all business rules, validation against a defined schema is crucial. They provide a Dart example of a strict parser that returns a specific error with a list of issues, allowing for targeted repairs in the next loop iteration.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
