Renting Out One T4 to Three Customers: Gemma 4, vLLM Multi-LoRA and a Rust Gateway

This article presents a guide to renting out a single NVIDIA T4 NVIDIA graphics processing unit (GPU) on a Google Cloud virtual machine (VM) to multiple small businesses. The system featured employs Gemma 4 E2B, powered by vLLM, and a lightweight Rust gateway that manages access and billing for the shared GPU.

The demonstration reveals that a single T4 struggles when several customers simultaneously submit large prompts, causing latency to increase dramatically. However, vLLM’s priority scheduling capability effectively minimizes this bottleneck, bringing the response time back to approximately 6.52 seconds.

A key aspect of this project is the isolation of each customer’s prompt and responses. This is achieved through keys, fine-tuned adapters, prefix caches, and a ledger that accurately matches vLLM’s token counters. In terms of billing, customers are charged based on GPU time, but a significant portion of this cost is offset by their token usage. For instance, a customer requesting lengthy prompts pays 27.2% of the VM’s cost for just 22.0% of the total tokens processed.