{
  "id": 16056497,
  "title": "The Number I'll Cite Next Year Won't Come From the Paper",
  "url": "https://urgent.news/2026/10/22/the-number-ill-cite-next-year-wont-come-from-the-paper",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-22T14:37:48.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/o96a/the-number-ill-cite-next-year-wont-come-from-the-paper-33ik"
  },
  "original_language": "en",
  "account": "A week lost in March due to a flat reward curve has sparked concerns about the credibility of headline numbers in reinforcement learning (RL) post-training papers. The author blames various factors such as data, learning rate, GPU, and configuration changes, but emphasizes that the real issue is the lack of reproducibility. RL post-training is more susceptible to failure compared to supervised fine-tuning (SFT) because the reward becomes the ground truth, making it easier for the model to be misled. The author argues that the only reliable way to detect reward hacking is to have an independent party run the same pipeline and verify the results. They propose that within the next eighteen months, the label \"independently reproduced\" will become a key citation for RL papers, rather than just the headline number. This change will be driven by model cards, leaderboards, and release posts that emphasize the importance of independent reproduction. The author suggests that labs should allocate a small portion of their training budget to fund outside runs, ensuring that the label carries weight. To make the label meaningful, the author outlines the necessary artifacts that should be made available, such as the evaluation harness version, reward model checkpoint, prompt, reward model snapshot date, seeds, data order, and a baseline for comparison. Until labs adopt these practices, the author advises readers to search for second implementations and reproduce the results before relying on the headline numbers.",
  "summary": "I lost a week in March to a flat reward curve. Not a dramatic failure. Loss went down, reward went up, and the model got quietly worse at the thing I actually cared about. I blamed my data. I blamed the learning rate. I blamed the GPU. Then I diffed my config against the appendix and found three differences: a different chat template, a reward model checkpoint two versions behind, and a KL…",
  "key_points": [
    "Lack of reproducibility concerns credibility of RL post-training numbers",
    "Independent reproduction proposed as reliable detection method",
    "Labs to fund outside runs, make artifacts available for verification"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}