{
  "id": 20780028,
  "title": "Your GPU is busy computing padding: cost lessons from fine-tuning on rented cards",
  "url": "https://urgent.news/2026/11/19/your-gpu-is-busy-computing-padding-cost-lessons-from-fine-tuning-on",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-19T17:56:48.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/servin_osmanov/your-gpu-is-busy-computing-padding-cost-lessons-from-fine-tuning-on-rented-cards-265k"
  },
  "original_language": "en",
  "account": "This article outlines the author's experience and lessons learned while renting cloud GPUs to train speech and language models for a low-resource language called Crimean Tatar. The models required significant computational resources, as full fine-tuning needed 141 GB of memory or more. To accommodate this, the training process was run on rented machines, with costs accumulating over a month and a half to over $300.\n\nThe author highlights several key lessons learned during the process:\n\n1. Estimate costs based on padded tokens, not content: In the study, the median example was 20 tokens long, but each example was padded to a sequence length of 1024 tokens. Padding did not affect the result but increased the bill, as the GPU still processed the full sequence and most computations were discarded. The author's estimate was based on content, expecting 59 million tokens, while the actual GPU processed 1.17 billion positions.\n\n2. Checkpoint by wall-clock time, not step count: The author saved full checkpoints every 400 steps to a network volume, which led to inefficient GPU utilization. A better approach is to set checkpoint frequency based on hours of work, considering the cost of an hour against the risk of losing work. Checkpoints should be saved to local disk for faster access.\n\n3. Match the GPU workload to available resources: The author found that two data loader workers preparing batches on a 128-core machine were faster than the GPU could process them. The GPU would complete a batch quicker than the loader could prepare the next one, causing idle time. The author now recommends running around a quarter of virtual CPUs as workers with the GPU, and to check GPU utilization within the first ten minutes of training. If utilization is low, there is an issue that needs to be addressed.\n\n4. Thoroughly test the entire pipeline: The author discovered that the training pipeline had passed a smoke test at home, but this test did not reflect the actual throughput. They completed five different machines in a row and each one failed before finishing. Some crashes were due to the model running out of memory, while others were caused by the system watchdog mistakenly detecting a failed run. The author emphasizes the importance of testing the entire pipeline to avoid unexpected issues during training.\n\n5. Rehearse the whole training process: After incorporating the previous lessons, the author started a second attempt and ran through five machines in a row. Each machine died before completing stage 2. The author believes that rehearsing the entire process, including packing and checkpointing, would have helped identify potential problems before they caused significant delays or financial losses.",
  "summary": "In the second post of this series I promised to write up how I rented cloud GPUs and burned money on them without doing the maths. In the third I said the promise still stood. Here it is. Some context first. I'm building speech and language models for Crimean Tatar, a low-resource language ( why , and where the data came from ). ASR and TTS train fine on my 24 GB card at home. Full fine-tuning of…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}