What one month of public benchmarks says about my text-to-SQL access layer
schemagate is an open-source layer for text-to-SQL that starts from one question: what is this caller allowed to see? It shows the model only the tables the current user may read, before any SQL is written, reads the permissions from your database's own grants, and refuses a query that touches anything else before it reaches the database. This month I ran it on public benchmarks. Every number…
In the past month, open-source text-to-SQL layer schemagate was put through its paces on various public benchmarks. The key takeaways come directly from the benchmark evaluators' reports, with all figures sourced from those reports.
Schemagate's access control mechanism, which restricts queries based on user permissions, outperformed a comparable setup reported in a SIGMOD 2027 paper. It achieved a 90.1 AC-F1 score, compared to the paper's 88.1 for the same model. Spider-ACL, another benchmark, recorded a refusal F1 score of 0.984 over 19,624 question-and-role pairs, a marked improvement over the previous best of 0.933.
When tested on the BIRD development set of 1,534 questions, schemagate demonstrated a 77.1% execution accuracy using a single model call per question, surpassing previous leaders. The Mini-Dev benchmark, covering SQLite, PostgreSQL, and MySQL, scored 81.0 / 72.6 / 72.4, outperforming current leaders.
The hidden test set, submitted for review, showed promising results. On Spider 2.0 SQLite questions (135), schemagate achieved a 71.9% Pass@1 rate with a single call per question, significantly higher than the 57.8% reported by the best published agent for the same subset.
Even when challenged with Oracle UniQL, schemagate performed admirably on the first try, achieving a 64.5% accuracy. The previous best published baseline for Oracle was 49.6%. Oracle dialect-specific prompt rules were crucial to this success.
The adoption of schemagate has been steady, with around 10,000 downloads on PyPI so far (PyPI statistics exclude mirrors). The project is licensed under Apache-2.0 and supports multiple databases—including Oracle, PostgreSQL, SQL Server, MySQL, and SQLite—as an MCP server, LangChain retriever, or CLI. To use schemagate, simply install it via pip with the command 'pip install schemagate'. For more details and to explore the full set of caveats, visit the benchmarks page on GitHub.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
