I tried to teach a small AI model a calculus textbook. The first thing I had to fix was how I was keeping score.
Five weeks, one free GPU, and a lesson about measuring before celebrating. One of my training runs made the model 19 points worse at calculus. My scorer said it had lost 3. That gap is really what this article is about. For a while, my numbers told me one story and the truth was another. Here's the whole thing: what I built, where it went wrong, and what I'd do differently next time. Why I tried…
In an experiment where one tried to train a small AI model on a calculus textbook, initial challenges arose in assessing the model's performance. The first attempt to evaluate the model resulted in it performing 19 points worse in calculus. The author, who was attempting this experiment, realized that the evaluation criteria were the root of the problem.
The model had learned to format its answers in a manner that met the scorer's expectations, but it did not gain a comprehensive understanding of calculus. The model's training process involved converting the textbook into clean text, which took around 550,000 tokens. However, the model only provided verified solutions after attempting each problem multiple times.
The author tried various training methods, including supervised fine-tuning and reinforcement learning, but saw only marginal improvements in accuracy. A significant issue arose when the scoring method was flawed, as it falsely credited the model for formatting improvements rather than actual understanding of calculus. This mistake led to misleading results, both in positive and negative directions.
To resolve this issue, the author rebuilt the evaluation process from scratch, ensuring that the model's answers were verified against the answer key.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
