A Better FP4 Gradient Quantizer That Training Couldn't Notice
Four-bit training is the current frontier of making LLM pretraining cheaper. NVIDIA's Blackwell GPUs do 4-bit matrix math several times faster than 16-bit, but 4 bits means every number gets rounded to one of 16 values. The NVFP4 format's menu is exactly {0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6} , stretched by one scale factor per block of 16 numbers. Choosing that scale is most of the game. The…
The article discusses a new gradient quantizer called FP4, which promises to make training large language models (LLMs) more efficient. NVIDIA's Blackwell GPUs can perform 4-bit matrix math significantly faster than 16-bit, but this comes at the cost of rounding numbers to one of 16 values. The NVFP4 format offers eight specific scales between 0 and ±6, and choosing the right scale is crucial for minimizing rounding errors.
The article's author discovered a scale rule that outperforms the current state-of-the-art estimator, MS-EDEN, by 14% on average. However, the improvement in training loss was not noticeable because the batch sizes used in the tests were too small to reveal the difference. The author emphasizes that the batch size where the improvement becomes visible is the one used in practical training scenarios.
The code for the quantizer is not publicly available, so the article provides the numbers and results instead. The article also explores the limitations of error feedback techniques when applied to Adam optimizer, revealing that they can actually worsen performance in certain situations.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
