Show "can't verify" instead of a wrong number: handling declines in an AI data app

When an AI data application attempts to answer a user's query, it provides a response that appears correct and plausible. However, a week later, a finance team member may discover the answer is inaccurate. The recommended solution is not to adjust the prompt but to implement a simpler approach. If the system cannot back up its answer, it should clearly indicate that it cannot verify the information.

This post explains how to implement this solution using Python and Node, utilizing FactQL. FactQL first runs the query before providing an answer. When it successfully returns results, the output includes the query executed, the data rows, and a clear explanation. For instance, when asking about the total revenue by region, the system returns an explanation stating that the North region leads with $43,148, followed by the South region with $41,223.

However, when the system cannot fulfill the query, it declines instead of guessing the answer. This decline status is still an HTTP 200 response, but it includes a decline_reason indicating why the system couldn't provide a safe SQL query. The explanation in this case reads, "I couldn't find data that answers this. Try naming the table or column you mean."

An important aspect is that the error handling code must check the result field before displaying any numbers. If a decline occurs, the explanation should be shown to the user, written in a way that they can understand and act upon. The solution advises against retrying the same question since it will likely receive the same incorrect answer.

Instead, retries should only be made for HTTP 429 (too many requests), 500, and 503 (server errors), after a Retry-After delay.