How Fast Can a 421M-Parameter Decision Model Run? I Benchmarked Laya Across NVIDIA GPUs

A 421-million-parameter decision model, known as Laya, can run at impressive speeds when tested across various NVIDIA GPUs. An independent benchmark measured the model's performance, focusing on sustained throughput rather than raw numbers. The benchmark used a frozen sample of 1,000 public SAM.gov contract-opportunity notices and asked the same three questions—choice, score, and yes/no—for each request.

This allowed reporting throughput as decisions per second rather than requests per second. The benchmark tested three GPUs: RTX 2000 Ada, RTX PRO 5000, RTX PRO 6000, and H100 NVL, along with different serving backends and precisions. The results showed that on the H100 NVL, Laya achieved 175 decisions per second with a p99 latency of around 91 milliseconds.

This translates to 15.1 million decisions per day while staying within the p99 ≤ 130 ms latency budget. The benchmark also revealed several key insights about serving small models. First, the latency knee is sharp, meaning that once a GPU reaches its sustainable capacity, its performance drops significantly. Second, TensorRT is not always the best choice for fixed-shape throughput; torch.compile mode, which automatically optimizes the model, often outperforms it.

Third, using separate MIG slices on the H100 NVL was not the most efficient approach for this workload. Finally, the benchmark demonstrated that a 10-million-decision day can be handled by a single RTX PRO 6000 GPU with zero errors and a p99 latency of 111 milliseconds. However, to remain below the 130 ms latency budget continuously, the deployment would need roughly 25% headroom at that volume.

Overall, these findings provide valuable insights into serving small decision models and highlight the importance of considering latency, throughput, and accuracy when making serving decisions.