LLM Returns Invalid JSON Once in a While: Structured Outputs vs Validate-and-Retry

When parsing JSON produced by an LLM occasionally fails, three main issues can be the root cause: truncation, extraneous text or code formatting around the JSON, and schema mismatches between the model output and expected structure. Structured outputs from providers help eliminate the latter two causes for certain schemas. To handle these failures, a validation-and-retry approach with a typed model is recommended, complemented by a logging mechanism for raw responses and stop reasons.

The most common error messages include JSONDecodeError indicating improper JSON structure, Unterminated string errors, and Pydantic validation errors pointing to specific issues like priority values exceeding limits. These errors have distinct fixes: removing code fences or brackets, adjusting schema limits, and enforcing strict type validation on the provider side.

Before modifying prompts, it's crucial to log raw output and stop reasons for each failure. Often, the cause is truncation, which cannot be resolved by prompt adjustments alone. The process begins by validating incoming data with a typed model. A function extracts the outermost JSON object, handling potential truncation or nested objects. Another function then attempts parsing using the model, catching validation errors and feeding them back to the model for correction.

The retry mechanism caps attempts to three to prevent excessive costs, and the assistant role provides the corrected JSON if validation fails. This approach ensures precise error feedback, minimizing manual intervention and optimizing LLM interactions in production environments.