Every text-to-SQL benchmark score you've seen was measured without access control
Spider, BIRD, LiveSQLBench. If you have evaluated a text-to-SQL system in the last five years you have quoted a number from one of them. All three ask the same question: given a schema and an English question, does the system produce SQL that returns the right rows? None of them ask who is asking. Every score you have seen was produced by a system with unrestricted read access to the entire…
Wire report analyzing a new benchmark for text-to-SQL systems that measures their performance under role-based access control (RBAC) restrictions. The benchmark, called Benchmarking Text-to-SQL under RBAC, adds role annotations and policies to three existing text-to-SQL benchmarks.
Key findings:
- High-performing text-to-SQL systems, especially open-weight LLMs, show sharp performance degradation when RBAC constraints are applied. This is because these systems frequently violate access policies, resulting in "RBAC-rejected successes" - SQL queries that are technically correct but violate the access policy.
- The problem arises because the evaluation metrics used by benchmarks like Spider and BIRD do not account for access control information. These metrics score a query as correct if it returns the right rows, without considering whether the user is authorized to access those rows.
- The issue stems from the fact that early steps in an NL2SQL pipeline, like selecting which tables to query, do not have access control information. As a result, the system may select a table to query that the user is not authorized to access, leading to a correct query being rejected.
- The paper does not propose a solution, but highlights the need for access control to be incorporated earlier in the NL2SQL pipeline, before the model writes any SQL. This would require the system to have knowledge of the user's role and access permissions when selecting the tables to query.
- The authors emphasize the importance of distinguishing between a "no records found" result and a "query denied due to access restrictions" result, as the latter reveals sensitive information about the database schema.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
