Your LLM returns JSON that crashes json.loads about 1 time in 20. Here is how to handle it

Your LLM returns JSON that occasionally crashes json.loads, typically about once in every twenty attempts. This issue can arise when the JSON is surrounded by markdown code fences, preceded or followed by explanatory sentences, or contains trailing commas. To handle these problems, there is a straightforward, step-by-step approach using plain Python.

First, you need to strip away the usual wrapping. Import the necessary libraries, namely json and re. Define a function named extract_json that takes a string as input. Within this function, use the strip method to remove any extra whitespace. Next, utilize a regular expression to search for JSON code fences, such as ```json or { % endraw % } json ... { % raw % } ```.

If a fence is found, extract the contents between the fence and store it in a variable. Then, remove any leading or trailing whitespace from the extracted JSON.

In the second step, parse the extracted JSON and handle the common error of a trailing comma before the closing bracket or brace. This can be done by defining a new function called parse_llm_json, which takes a string as input. Call the previously defined extract_json function to obtain the raw JSON. Then, use a try-except block to attempt parsing the raw JSON using json.loads.

If a JSONDecodeError occurs, remove any trailing commas before the closing bracket or brace using a regular expression and sub method. Finally, parse the repaired JSON string again and return it.

The third step involves validating the shape of the parsed JSON, ensuring it contains the correct keys and data types. Define a function named require that takes two dictionaries as input, one containing the actual data and another specifying the expected schema. Iterate through the schema dictionary and check if each key exists in the data dictionary.

If a key is missing, raise a KeyError with an appropriate error message. Similarly, check if the data type of each key matches the expected type in the schema dictionary. If a type mismatch is found, raise a TypeError with a descriptive error message. If the data passes all validations, return it.

Lastly, implement a retry mechanism to handle the occasional failure. Define a function named ask_json that takes four parameters: call_llm (a function to call the LLM), prompt (the question to ask the LLM), schema (the expected JSON schema), and retries (the number of retry attempts, defaulting to 1). Call the call_llm function with the provided prompt and store the response in a variable.

Use a for loop to iterate through the retry attempts, starting from 1. Within the loop, call the previously defined parse_llm_json function to parse the JSON response. Use the require function to validate the parsed JSON against the expected schema. If any ValueError, KeyError, or TypeError occurs during this process, check if the current attempt is equal to the number of retries.

If it is, re-raise the exception. Otherwise, call the call_llm function again with a modified prompt that includes the exact error message and request only the corrected JSON without any extra text.