Get Reliable JSON from an LLM in Python with Pydantic

In the source material, a small shop receives customer emails containing order information. A script is used to turn each email into data and feed it directly into a spreadsheet. The script initially works well for a week, but then begins producing incorrect JSON responses, such as adding chatty text, cutting off replies, and inventing order numbers that were not present in the email.

The guide builds a tested loop that asks for JSON, checks the answer, and requests a new answer when the response is not satisfactory. JSON is crucial for most APIs, making it important to get it right whenever a model feeds a program. There are four ways the JSON response can go wrong: it may not be JSON, the shape may be incorrect, a field may be missing or renamed, or the values may be wrong.

The guide explains how to handle these issues using Pydantic, a Python library for describing and checking data. The first step is to describe the data by creating a Python file called orders.py that defines a valid order structure and an extra check for values that do not appear in the email. The second step involves adding a helper function called check_grounded to the llm_json.py file, which asks a language model for JSON, checks the answer, and requests a new answer if it is incorrect.

The guide emphasizes the importance of validating the model's output against a schema, checking the answer against the source text, and failing safely after a few attempts.