Your LLM Returns JSON That Isn't JSON: A Robust Structured-Output Pipeline for Local Models
Your LLM Returns JSON That Isn't JSON: A Robust Structured-Output Pipeline for Local Models You asked a local model for JSON. You got JSON. You json.loads() it and — JSONDecodeError: Expecting value. Because buried in the "JSON" was a code fence, three sentences of "Here is your result:", and a trailing comma no parser will forgive. If you've wired a local LLM into an agent, an ETL job, or a…
A robust pipeline for handling structured output from local models has been developed to address issues with JSON parsing. When a local model returns JSON, it may contain unexpected elements such as code fences or incorrect formatting that can cause JSONDecodeError. This article presents a solution that combines Ollama's schema-constrained decoding with a resilient parser, schema validation, and feedback-driven retries.
The first step is to pass a real JSON Schema instead of the string "json" when requesting JSON output from the model. The Python Ollama client can easily achieve this using Pydantic, a data validation library. By providing the schema in the format parameter, the model will be prevented from emitting invalid JSON. This is the recommended pattern for most straightforward schemas on a 7B-plus model.
However, this approach has limitations. It may not work with different local servers, older Ollama versions, or endpoints that only offer loose JSON mode. Even if the schema constrains the structure, it does not guarantee the correctness of the values. For instance, the model may hallucinate a number when the schema demands an integer.
To handle these edge cases, the article introduces a resilient parser that can handle hostile output. The json_repair library, a drop-in replacement for json.loads(), can fix issues like missing quotes, trailing commas, truncated values, and stray prose. It also supports schema-guided repair and strict mode for raising exceptions instead of repairing.
After parsing, the article emphasizes the importance of validation against the contract. If validation fails, the system should retry with the error fed back to the model, not a blind re-roll. A maximum of one to three attempts is recommended, after which the system should fail loudly and keep the raw output for debugging purposes.
The article provides a self-contained drop-in pipeline that works directly against the Ollama chat API, demonstrating all the steps involved. It uses the ollama library for making requests to the local model, pydantic for schema validation, and json_repair for handling hostile output. This pipeline ensures that the structured output from local models is reliable and can be trusted for further processing.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
