{
  "id": 5861649,
  "title": "Backtest 63.7%, production AUC 0.51 - a postmortem",
  "url": "https://urgent.news/2026/09/05/backtest-63-7-production-auc-0-51-a-postmortem",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-05T23:07:00.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/khavel/backtest-637-production-auc-051-a-postmortem-2alc"
  },
  "original_language": "en",
  "account": "A once-promising model that predicted NBA and WNBA player props hit a surprising setback. Built with seven independent blocks, it achieved a 63.7% hit rate and 0.56 AUC in offline testing. However, when deployed in production, its performance degraded to a coin flip (AUC of 0.51) and inverted top-tier rankings. The root cause was a missing calibration snapshot with an undocumented export query, leading to a train-serve mismatch between the training and serving populations. To prevent such issues, the model is now parked, with provenance and data provenance mandatory for any dataset influencing decisions. Any model replacement now requires forward data validation and monitoring of calibration, Brier score, log loss, AUC, and per-tier hit rates in a trailing window. The new approach includes publishing model misses alongside hits in a public record, promoting transparency and accountability.",
  "summary": "Originally published at nbaproplab.com/learn/model-postmortem . A model that looks excellent offline and random in production is usually blamed on the serving code. Ours was not: the two computations matched to 2e-16. The cause was a calibration snapshot whose export query nobody wrote down. What we built We score NBA and WNBA player props through seven independent blocks: player form, matchup,…",
  "key_points": [
    "Model achieved 63.7% hit rate and 0.56 AUC in offline testing",
    "Performance degraded to coin flip (AUC of 0.51) in production",
    "Missing calibration snapshot caused train-serve mismatch"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}