{
  "id": 16269864,
  "title": "What one month of public benchmarks says about my text-to-SQL access layer",
  "url": "https://urgent.news/2026/10/24/what-one-month-of-public-benchmarks-says-about-my-text-to-sql-access",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-24T03:20:24.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/ashish_sinha_5241c7673d93/what-one-month-of-public-benchmarks-says-about-my-text-to-sql-access-layer-4n91"
  },
  "original_language": "en",
  "account": "In the past month, open-source text-to-SQL layer schemagate was put through its paces on various public benchmarks. The key takeaways come directly from the benchmark evaluators' reports, with all figures sourced from those reports.\n\nSchemagate's access control mechanism, which restricts queries based on user permissions, outperformed a comparable setup reported in a SIGMOD 2027 paper. It achieved a 90.1 AC-F1 score, compared to the paper's 88.1 for the same model. Spider-ACL, another benchmark, recorded a refusal F1 score of 0.984 over 19,624 question-and-role pairs, a marked improvement over the previous best of 0.933.\n\nWhen tested on the BIRD development set of 1,534 questions, schemagate demonstrated a 77.1% execution accuracy using a single model call per question, surpassing previous leaders. The Mini-Dev benchmark, covering SQLite, PostgreSQL, and MySQL, scored 81.0 / 72.6 / 72.4, outperforming current leaders.\n\nThe hidden test set, submitted for review, showed promising results. On Spider 2.0 SQLite questions (135), schemagate achieved a 71.9% Pass@1 rate with a single call per question, significantly higher than the 57.8% reported by the best published agent for the same subset.\n\nEven when challenged with Oracle UniQL, schemagate performed admirably on the first try, achieving a 64.5% accuracy. The previous best published baseline for Oracle was 49.6%. Oracle dialect-specific prompt rules were crucial to this success.\n\nThe adoption of schemagate has been steady, with around 10,000 downloads on PyPI so far (PyPI statistics exclude mirrors). The project is licensed under Apache-2.0 and supports multiple databases—including Oracle, PostgreSQL, SQL Server, MySQL, and SQLite—as an MCP server, LangChain retriever, or CLI. To use schemagate, simply install it via pip with the command 'pip install schemagate'. For more details and to explore the full set of caveats, visit the benchmarks page on GitHub.",
  "summary": "schemagate is an open-source layer for text-to-SQL that starts from one question: what is this caller allowed to see? It shows the model only the tables the current user may read, before any SQL is written, reads the permissions from your database's own grants, and refuses a query that touches anything else before it reaches the database. This month I ran it on public benchmarks. Every number…",
  "key_points": [
    "Schemagate's access control outperformed SIGMOD 2027 paper by 90.1 AC-F1 score",
    "Spider-ACL refusal F1 score improved to 0.984 over 19,624 pairs",
    "Schemagate achieved 77.1% execution accuracy on BIRD development set"
  ],
  "editors_take": "Superior accuracy across multiple database dialects and access control benchmarks establishes schemagate as a new performance leader for developers seeking reliable text-to-SQL integration within their existing data workflows.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}