The sm_120 shared-memory cliff: why FP8 KV cache crashes vLLM on workstation Blackwell

The shared-memory issue affecting vLLM on workstation Blackwell cards stems from the fp8 KV cache crashing when the GPU reaches its shared memory limit. DeepSeek-family models such as MLA models running on RTX PRO 6000 or any workstation or consumer Blackwell card may experience this issue on startup, resulting in "triton.runtime.errors.OutOfResources: out of resource: shared memory" error.

Reducing block sizes or the number of stages may help alleviate the problem. The hardware fact behind this issue is that datacenter Blackwell parts advertise a larger per-block shared-memory budget compared to workstation and consumer silicon. On an RTX PRO 6000 Blackwell (sm_120) GPU, the shared memory per block is capped at 101,376 bytes.

When configuring the kernel for vLLM, launching with a specific tile configuration and sweeping the `num_stages` parameter can help determine the cause of the crash. It was found that with bf16 KV, adding an extra pipeline stage costs about 4 KB, while with fp8 KV, a single stage costs about 18 KB, and stage two consumes the full per-SM budget, exceeding the per-block capacity by 1,024 bytes.

The recommended solution to this shared-memory cliff is to compute the tile's actual shared-memory requirement and compare it to the shared_memory_per_block_optin for the specific device being used. This approach handles various Blackwell cards and similar hardware without requiring a special case for each chip.