{
  "id": 3275836,
  "title": "A Better FP4 Gradient Quantizer That Training Couldn't Notice",
  "url": "https://urgent.news/2026/08/25/a-better-fp4-gradient-quantizer-that-training-couldnt-notice",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-25T14:00:00.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/megapixel99/a-better-fp4-gradient-quantizer-that-training-couldnt-notice-379l"
  },
  "original_language": "en",
  "account": "The article discusses a new gradient quantizer called FP4, which promises to make training large language models (LLMs) more efficient. NVIDIA's Blackwell GPUs can perform 4-bit matrix math significantly faster than 16-bit, but this comes at the cost of rounding numbers to one of 16 values. The NVFP4 format offers eight specific scales between 0 and ±6, and choosing the right scale is crucial for minimizing rounding errors. The article's author discovered a scale rule that outperforms the current state-of-the-art estimator, MS-EDEN, by 14% on average. However, the improvement in training loss was not noticeable because the batch sizes used in the tests were too small to reveal the difference. The author emphasizes that the batch size where the improvement becomes visible is the one used in practical training scenarios. The code for the quantizer is not publicly available, so the article provides the numbers and results instead. The article also explores the limitations of error feedback techniques when applied to Adam optimizer, revealing that they can actually worsen performance in certain situations.",
  "summary": "Four-bit training is the current frontier of making LLM pretraining cheaper. NVIDIA's Blackwell GPUs do 4-bit matrix math several times faster than 16-bit, but 4 bits means every number gets rounded to one of 16 values. The NVFP4 format's menu is exactly {0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6} , stretched by one scale factor per block of 16 numbers. Choosing that scale is most of the game. The…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}