Advanced GPU Optimization: How to tech an LLM with CUDA and ROCm? - Part 5 (Final Part)

Welcome back, you ambitious coder! You've conquered Part 4, implemented Flash Attention, and harnessed the power of FP8 precision. Now, let's tackle the final piece of the puzzle: implementing Mixture of Experts (MoE) routing and expert parallelism on the GPU.

MoE is a game-changer for large language models, enabling them to scale to trillions of parameters without causing compute costs to skyrocket. The trick lies in having numerous expert Feed-Forward Neural Networks (FFNs), but only activating two of them for each token.

To achieve this, we'll focus on two key components: the Router Kernel and Expert Parallelism with All-to-All Communication.

1.1 The Router Kernel (Top-k Gating)

First, we need a CUDA kernel to compute the routing scores for each token and select the top-2 experts. The kernel takes the token embeddings as input, computes the logits for each expert, and tracks the indices and weights of the top-2 experts using a manual reduction.

The Router Kernel iterates through each token, computes the scores using the router weights, and updates the max1 and max2 variables to maintain the top-2 scores and indices. Then, it applies the softmax function only to the top-2 scores, ensuring a sparse softmax operation.

1.2 Expert Parallelism with All-to-All Communication

Unlike dense models that used All-Reduce communication, MoE requires All-to-All communication since different GPUs host different experts. Tokens must be sent to the GPU that owns the expert responsible for processing them.

To implement this, we'll use NCCL or RCCL for sending and receiving tokens between GPUs. The process involves two main steps:

1. Token dispatch: Build a send buffer per rank based on the routing decisions, ensuring that each GPU sends its tokens to the GPU that owns the assigned expert. After sending, the GPU receives the tokens for its local experts.

2. Local expert execution: Run the local expert FFNs on the received tokens using a kernel. This step is executed locally on each GPU, allowing for parallel execution of experts.

By combining the Router Kernel with All-to-All communication, we can efficiently implement MoE routing and expert parallelism on GPUs, enabling large language models to scale efficiently without running into VRAM limitations.