I Got 28 TPS Out of Free Kaggle GPUs. Here's What It Took.

I Got 28 TPS Out of Free Kaggle GPUs. Here's What It Took.

A 7-billion parameter model in FP16 needs around 15 gigabytes of VRAM. A single Kaggle T4 has 16GB, barely enough for the model and a key-value cache. To run such a large model for free, developers split the model across two machines, with each handling different layers and the language model head. The machines communicate over the internet, which adds latency to the process.

The initial throughput was only 4.92 tokens per second (TPS). However, researchers discovered speculative decoding, which improved performance significantly. By generating a small number of token candidates locally and sending them all to the verifier in one round trip, the TPS increased to 14.3. This method is much faster than the traditional sequential token generation and drastically reduces the time spent waiting for network responses.

Despite the improvements, the developers still hit a bottleneck. The draft model, a smaller version of the 7B model, was running separate forward passes through the 0.5B model, causing the GPU to idle for a significant portion of each round. The Python code responsible for issuing CUDA kernel launches was taking longer than the actual GPU computation, leading to inefficient utilization of the GPU resources.

To address this issue, the developers made changes using CUDA Graphs. CUDA Graphs allow capturing a sequence of GPU operations and replaying them as a single driver call, eliminating the need for multiple CUDA kernel launches. After implementing CUDA Graphs, draft generation time decreased from 112 milliseconds to just 25ms, resulting in a four-fold improvement in generation speed.

However, the developers encountered a problem when using CUDA Graphs - they started to get garbage output. This issue was caused by tensor reallocation during replay, which broke the captured GPU memory addresses. To fix this, they switched to StaticCache instead of DynamicCache, performed in-place tensor mutation, explicitly updated position_ids, and implemented in-place KV rewind. These changes ensured that the graph captured valid addresses throughout the generation process, eliminating the garbage output.

The final results showed incredible performance: 28.10 TPS on Qwen2.5-7B across two Kaggle T4s over WAN. This achievement demonstrates that with the right optimizations, it's possible to run large models on free GPUs and achieve impressive results. The developers learned that profiling matters more than intuition, and focusing on the Python kernel launch overhead led to a significant boost in performance.