{
  "id": 20645981,
  "title": "Why your fine-tuning job takes longer than the FLOPs say",
  "url": "https://urgent.news/2026/11/18/why-your-fine-tuning-job-takes-longer-than-the-flops-say",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-18T18:53:43.000Z",
  "source": {
    "name": "Hugging Face",
    "slug": "hugging-face",
    "url": "https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time"
  },
  "original_language": "en",
  "account": "Fine-tuning large language models on shared GPU clusters often takes far longer than simple FLOP-based estimates predict. Researchers analyzed 272 production runs from NVIDIA H200 GPUs, comparing analytical estimates against predictors derived from past job data. The standard FLOP calculation explains only 31% of the variance in observed job durations, with a mean absolute error of 12.41 hours. In contrast, a CatBoost regressor that incorporates more training-specific signals achieved an R² of 0.98 and a mean absolute error of just 1.71 hours. The key factors influencing duration include training regime, evaluation cadence, per-GPU load, and fine-tuning specifics, which the model learns from historical patterns. Other significant variables include the number of evaluation data samples processed, the count of trainable parameters, batch evaluation size, and the total number of GPUs and training samples. While the FLOP estimate provides a rough baseline, end-to-end duration requires accounting for additional overheads like setup, periodic evaluations, and checkpoint uploads. These factors can cause jobs with similar nominal compute requirements to finish hours apart due to competition for network and storage resources on multi-tenant clusters. Overall, the CatBoost model captures the complex interplay of job-specific and environmental factors that drive actual fine-tuning times, offering a much more accurate prediction than simple FLOP calculations.",
  "summary": null,
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}