Fast Attention Kernels and Operator Fusion
Overview This post is divided into four parts; they are: • Operator Fusion • FlashAttention-Style Kernels • Paged Attention Kernels • Fusion and CUDA Graphs with torch.
We haven't written up this one. Machine Learning Mastery has the full story — the link below goes straight to it.
Source: machinelearningmastery.com