FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper
Transformer prefill reuses weights across tokens and is dominated by matrix multiplication and activation data movement; single-token decode has limited reuse and is dominated by weight and KV-cache reads. We present FUTRANSFORMER, a CUDA implementation of a Llama-style Transformer block for NVIDIA Hopper GPUs. It applies CODA's GEMM-plus-epilogue approach to inference, adding direct KV-cache…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
