Chip Huyen explains how to cut inference costs without new hardware
Last October, the P99 conference — the online gathering for developers focused on high-performance, low-latency applications — featured a cracking The post Chip Huyen explains how to cut inference costs without new hardware appeared first on The New Stack .
In October 2025, Chip Huyen, author of the AI Engineering book, delivered a keynote at the P99 conference about optimizing inference costs for frontier models. Training a frontier model is a one-time cost, while inference incurs ongoing expenses. Huyen noted that the compute ratio between training and inference ranges from 1:10 to 1:100. This realization highlights the importance of reducing inference costs to recover training investments and sustain profitability.
To address this issue, Huyen spent months researching optimization techniques for LLM inference. She distilled her findings into a 30-minute presentation for the conference. Huyen suggested focusing on three key metrics to measure inference performance: Time to First Token (TTFT), Time Per Output Token (TPOT), and End-to-End Latency.
TTFT measures the time it takes for the user to see the first token, TPOT is the average time between consecutive tokens, and End-to-End Latency combines TTFT and TPOT with the number of output tokens minus one.
When choosing metrics to prioritize, Huyen emphasized the importance of considering user needs and goodput alongside throughput. Goodput measures the requests that meet target performance metrics. Huyen provided an example: if an app targets 200 ms TTFT and 100 ms TPOT, processing 10 requests per minute, but only three requests meet both criteria, it indicates potential areas for optimization.
Huyen discussed three main approaches to optimize LLM inference: hardware, model, and service. While hardware optimizations are intellectually interesting, they are not relevant for most people since changing hardware is beyond their control. Replicating machines or adding more machines can be costly and complicated, especially with varying machine sizes and model architectures. Therefore, Huyen focused on optimizing the model and the service layer.
Model optimizations involve altering the model weights to improve outputs. Quantization is one such technique that reduces the precision used to store weights and activations. For instance, lowering precision from 32 bits to 8 bits can reduce memory requirements, lower costs, and improve speed. However, quantization may result in minor quality degradation. Huyen observed that many companies now run models at reduced precision to balance performance and efficiency.
Written by urgent.news from The New Stack's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
