Mastering Low-Precision AI: FP8 and FP4 Support Across Frameworks in Mid-2026

In mid-2026, the adoption of FP8 and FP4 precision formats has become crucial for enhancing the efficiency of large-scale AI training and inference processes. FP8 supports two formats: E4M3 for improved precision in activations and weights, and E5M2 for a broader dynamic range in gradients. NVIDIA's NVFP4 takes this further by utilizing 4-bit values and micro-block scaling, which offers 16-element shared FP8 scales and tensor-level scaling.

These advancements significantly reduce memory utilization and boost throughput on modern GPUs compared to traditional BF16 or FP16 formats, enabling the training and deployment of larger models on the same hardware.

The advantages are evident: FP8 provides approximately 2 times memory savings, while NVFP4 can achieve up to 3.5 times memory reduction. Additionally, NVFP4 enhances Tensor Core performance and improves energy efficiency. However, these benefits come with trade-offs, including a reduced numerical range and precision, which may result in accuracy loss or instability if not properly managed.

Techniques such as delayed scaling, stochastic rounding, Hadamard transforms, and selective quantization are employed to mitigate these issues, often maintaining accuracy within 1-2% of higher-precision baselines on real-world workloads.

Research on these precision formats has progressed rapidly from the foundational 2022 FP8 paper to significant studies in 2025 demonstrating the stability of FP4 pre-training for multi-billion-parameter models. Hardware support for FP8 has matured on Hopper GPUs and reached its peak on Blackwell with native NVFP4 and MXFP8 acceleration.

Among the major frameworks, PyTorch currently leads with native float8 dtypes, Transformer Engine for production training, and TorchAO for optimized inference. JAX supports FP8 through Transformer Engine, while TensorFlow/Keras offers simpler quantize-to-FP8 options but leans more on TensorRT for high performance. Libraries like bitsandbytes are useful for complementary 4-bit memory savings.

Practical adoption of these precision formats is strong for both training and inference, especially when teams follow proven recipes, monitor scaling factors, and prototype on smaller models. Workarounds for any remaining limitations include casting unsupported operations to higher precision or utilizing selective quantization. FP8 is now considered production-ready for most teams, while NVFP4 is becoming increasingly practical on Blackwell hardware for maximum efficiency.

For those interested in exploring how specific numbers behave in FP8 or BF16, a free converter is available at https://www.bestgpusforai.com/calculators/number-to-GPU-float-converter. For further insights, real-world tips, and discussions on AI programming hardware, the community at https://www.reddit.com/r/AIProgrammingHardware welcomes contributions.