How to build a tiny 1.5B text-to-SQL model that beats a 7B
I wanted to build something with an LLM using my own hands. Not wire an API into a wrapper, but take a base model, train it, measure it, break it, and serve it over HTTP. Why? Because I wanted to get my hands dirty working with an LLM, making and breaking things. Text to SQL fits exactly. You ask a question in plain English, the system writes SQL, and you can run the query to find out whether it…
The author describes the process of building a tiny 1.5B text-to-SQL model that outperforms a larger 7B model. The goal was to get hands-on experience with LLMs by training, measuring, and serving a model that could answer questions about a database by generating SQL queries. The model was based on Qwen2.5-0.5B, trained using LoRA adapters, followed by supervised fine-tuning on question and query pairs, and reinforcement learning with GRPO.
The author initially trained the model three times in a row on Qwen2.5-0.5B, with each stage improving slightly. However, the model's accuracy remained low at 6.4%. The author discovered that the issue was not with the model itself but with the evaluation process. There was no held-out evaluation, meaning the model had only been tested on data it had trained on, resulting in inflated accuracy scores.
After rebuilding the model with a held-out evaluation, the accuracy improved significantly to 44.6% on real schemas and 49.7% with a proper RL reward. Switching to Qwen2.5-Coder-1.5B further boosted the accuracy to 68.1%. The author concludes that the 1.5B model is still inferior to the 7B model at one attempt, but it achieves the same level of performance by answering eight times and voting on the most common result, trading compute for the gap.
This demonstrates that taking a small model, focusing on a single task, and building a robust system around it can lead to impressive results.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
