38% of an analyst's questions get "no records found" when the data is right there
Your text-to-SQL agent has a failure mode that logs nothing, alerts nothing, and returns a confidently wrong answer. I finally put a number on how often it can happen, and on a small schema with an ordinary role split the number is 38.5%. The failure Someone asks a question whose answer lives in a table they are not allowed to read. The model is handed the whole schema, because that is what…
When an analyst poses a question, sometimes the database has no records to provide. In a small schema with an ordinary role split, 38.5% of the analyst's questions fail to return any records. This occurs when the data is present, but the analyst lacks permission to view that data. The model writes a correct query, but row-level security removes all rows.
The application receives an empty result set and reports that there are no records, even though the data exists. The issue arises because permission checks happen after generating the query, not before. A pipeline that scopes the schema by caller identity can detect these cases before any SQL is written. Building probe questions out of the tables' own words can help identify which tables are off-limits to the caller.
This measurement is not a measure of the model's performance, but rather a measure of whether the data is accessible.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
