Getting reliable JSON from LLMs: schemas, validation, retries and a deterministic fallback
To get reliable structured data from an LLM, use four layers : ask for a JSON Schema (and use the provider's structured-output or tool-calling mode where available), validate the result in code with a schema library such as Pydantic, retry once with the validation error included in the prompt, and fall back to a deterministic default if it still fails. Schema-constrained decoding fixes the shape,…
To obtain reliable structured data from language models (LLMs), employ four layers: request a JSON Schema (or utilize structured-output or tool-calling modes where available), validate the response using a schema library like Pydantic, retry once if validation fails, and implement a deterministic fallback if necessary. Schema-constrained decoding guarantees the shape of the data, but not its content; a valid object may still contain incorrect information.
Therefore, business rules must be checked in code. Plain prompting does not suffice, as models often wrap JSON in Markdown code fences, add sentences before it, omit required fields, append unwanted fields, return numbers as strings, or produce lists when a single item is expected. Such issues, if passed to downstream pipeline stages, become bugs that trace back to their origin.
Layer One: Provide the model with a schema. Most major APIs now support structured outputs through JSON Schema response formats or tool calling, where the tool's input schema defines the data structure. When the provider enforces decoding to the schema, malformed JSON is significantly reduced. However, maintain Layer Two nonetheless: you might switch providers or models, use a gateway like LiteLLM, or encounter a model that lacks strict mode support.
Layer Two: Validate the response in code. Define the data shape once and validate every response against it. For instance, in Python, you can define a Pydantic model named LeadSummary with fields for company name, industry, employee count, and a summary. The `LeadSummary.model_validate_json(text)` function attempts to parse the provided text into a typed object or raises a `ValidationError` specifying which field failed and why. Length limits and allowed values should be part of the schema, not comments in the prompt.
Layer Three: Retry with the validation error message. If validation fails, resend the error to the model, which usually corrects the mistake. The `extract()` function in the provided Python code demonstrates this approach. It prompts the LLM, attempts to validate the response, and if a `ValidationError` occurs, appends the error message and the original prompt to the next request. Retry attempts are limited to two to avoid unnecessary cost and latency.
Layer Four: Content validation and deterministic fallback. Schema validity does not guarantee correctness. Additional domain-specific checks must be performed. For example, ensure placeholders like {first_name} or [COMPANY] are not left unfilled, numbers originate from the original input rather than being invented, and dates adhere to logical constraints (e.g., an end date must follow a start date).
If these checks fail, return a deterministic fallback, such as a template, a "needs review" status, or null, which the caller can handle appropriately.
For monitoring purposes, log essential details for each call, including the model used, the prompt version, whether validation succeeded on the first attempt, any validation errors encountered, and if the fallback was invoked. The fallback rate serves as a critical health metric for LLM features. A rising fallback rate indicates that a model update or prompt modification has introduced a silent issue.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.