{
  "id": 682691,
  "title": "Scaling Laws: What Chinchilla Said and What Changed Since",
  "url": "https://urgent.news/2026/08/12/scaling-laws-what-chinchilla-said-and-what-changed-since",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-12T17:20:20.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/multigrid/scaling-laws-what-chinchilla-said-and-what-changed-since-5edp"
  },
  "original_language": "en",
  "account": "Chinchilla, a 2020 study, challenged the prevailing belief that adding more compute should always result in better language models. The researchers found that a better approach might be to scale both the number of parameters and the amount of training data together. Specifically, they concluded that roughly twenty training tokens should be allocated for every model parameter at the optimal level of compute usage. This leads to a ratio of about twenty training tokens per parameter for the most efficient model. A follow-up analysis by Besiroglu et al. in 2024 suggested that the original Chinchilla coefficients might not accurately reflect the data used in the study, but the general conclusion remains valid: smaller models trained extensively can outperform larger models that are only lightly trained. The key takeaway is that the cost of serving large models, which is proportional to their number of parameters, can become significantly more expensive than the initial training cost of smaller models that are trained more intensively. This has led to the emergence of small, heavily-overtrained models as a commercially viable size, as their per-token serving cost becomes much lower compared to larger models. However, it's important to note that the Chinchilla study focuses on pretraining loss and does not directly address the model's capabilities or how improved performance translates into practical applications. Additionally, factors such as data quality, filtering, and the use of synthetic training data can significantly influence the relationship between model size and performance.",
  "summary": "Chinchilla is quoted as a rule about how many tokens to train on. That is the part that stopped being followed. The reason it stopped is a different piece of arithmetic that nobody published as a headline, and it is short enough to do here. What a scaling law is An empirical relationship between resources and pretraining loss. Across many training runs at different sizes, loss falls as a power…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}