RTX 3090, RTX 5090, A100 and H100 on vLLM: the cheapest GPU flips with your answer time

The article compares the cost-effectiveness of four GPUs - RTX 3090, RTX 5090, A100 80GB PCIe, and H100 80GB SXM - when running the Qwen2.5-7B-Instruct model on vLLM. The cheapest option varies depending on the desired answer time.

For answers under 2 seconds, only the H100 can meet this requirement at $0.441 per 1 million output tokens. When allowing up to 3 seconds for answers, the H100 is $0.162, the RTX 5090 is $0.382, and the A100 is $0.803. In this scenario, the H100 is about 2.4 times cheaper than the RTX 5090.

As answer time increases to 4 seconds, the RTX 5090 becomes the cheapest at $0.120, followed by the H100 at $0.162, and the A100 at $0.243. At this point, the RTX 5090 is about 1.4 times cheaper than the H100.

With an answer time constraint of 8 seconds, the RTX 5090 is once again the most cost-effective at $0.050, more than double the price of the H100 at $0.099 and around 2 times cheaper than the A100 at $0.106.

Interestingly, when running all four GPUs with the maximum number of requests (64) in flight, the H100 remains the most affordable at $0.162, even though the RTX 5090 is significantly slower at 5.4 seconds (p50). Conversely, when using the maximum number of requests (64) and a fixed answer time, the H100 is the most expensive at $0.162, while the RTX 5090 is $0.082 and the A100 is $0.106.

The article emphasizes that the choice of GPU depends on the balance between cost and answer time, as the most cost-effective option changes based on the specific requirements. Additionally, the cost per token can fluctuate depending on the cloud provider's pricing, making it essential to consider different pricing structures when evaluating GPU performance.