My betting model looked great for 3,671 predictions. It never beat the bookmaker.

Over the course of 16 days, the sports-betting model meticulously recorded the predicted probabilities it generated prior to each match and then compared those predictions with the actual outcomes. At first glance, the model appeared promising: a Brier score of 0.1629, indicating it performed 30.8% better than randomly guessing the base rate.

The calibration error was just 3.0%, meaning the model's predictions aligned closely with the actual outcomes; it expected a 37.8% chance of an event happening, and that event occurred 38.2% of the time.

One might have been convinced that the model was working effectively. However, the key question in the world of betting is whether the model's predictions are superior to the bookmaker's odds. When the model's predictions were compared to the bookmaker's probabilities (after removing the bookmaker's vig), the results were inconclusive.

For the 510 predictions where the bookmaker's probability was available, the model's Brier score was 0.1874, slightly higher (by +0.00105) than the bookmaker's score of 0.1863. The paired difference was within a 95% confidence interval of [−0.00063, +0.00276], which is too small to confidently claim the model outperformed the bookmaker.

In essence, the model merely matched the bookmaker's odds; it didn't offer any edge. A more useful metric is the closeness of the model's probabilities to the bookmaker's odds. The median gap was 1.0 point, with a strong correlation of 0.9941, indicating the model's predictions were essentially a copy of the bookmaker's odds—with a slight adjustment.

Given that the model was constructed to merely follow the bookmaker's odds, it couldn't beat the bookmaker's margin by definition. The model's performance had been evident from the outset, before even a single match had taken place.

To ensure transparency and replicability, the author developed an open-source tool called "eval-pack," which requires no external dependencies and is written in plain TypeScript for Node.js version 22.6 and above. The tool mandates a comparison against a reference system—the bookmaker's odds—in order to generate any grading. The author encourages readers to verify the findings independently by cloning the provided GitHub repository, navigating to the "llm-eval-public/pack" directory, and running the "eval-pack" command.

The dataset used for the analysis is real and anonymized, containing no identifiable information about teams, players, operators, or individuals, and the methodology remains applicable to other artificial intelligence systems such as classifiers, RAG pipelines, judges, and agents. Ultimately, the question "Is the new model better?" is equivalent to asking "Does my model outperform the bookmaker?"—both hinge on the crucial comparison to an established reference.