Backtest 63.7%, production AUC 0.51 - a postmortem
Originally published at nbaproplab.com/learn/model-postmortem . A model that looks excellent offline and random in production is usually blamed on the serving code. Ours was not: the two computations matched to 2e-16. The cause was a calibration snapshot whose export query nobody wrote down. What we built We score NBA and WNBA player props through seven independent blocks: player form, matchup,…
A once-promising model that predicted NBA and WNBA player props hit a surprising setback. Built with seven independent blocks, it achieved a 63.7% hit rate and 0.56 AUC in offline testing. However, when deployed in production, its performance degraded to a coin flip (AUC of 0.51) and inverted top-tier rankings. The root cause was a missing calibration snapshot with an undocumented export query, leading to a train-serve mismatch between the training and serving populations.
To prevent such issues, the model is now parked, with provenance and data provenance mandatory for any dataset influencing decisions. Any model replacement now requires forward data validation and monitoring of calibration, Brier score, log loss, AUC, and per-tier hit rates in a trailing window. The new approach includes publishing model misses alongside hits in a public record, promoting transparency and accountability.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
