{
  "id": 1858217,
  "title": "Tokens per Second Benchmarks Explained: What You're Actually Measuring",
  "url": "https://urgent.news/2026/08/19/tokens-per-second-benchmarks-explained-what-youre-actually-measuring",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-19T03:51:59.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/apeder/tokens-per-second-benchmarks-explained-what-youre-actually-measuring-47hl"
  },
  "original_language": "en",
  "account": "Tokens per second (tok/s) is a crucial metric for measuring the performance of local large language models (LLMs). It represents the number of model-generated text fragments, called tokens, produced every second. However, tok/s is not as straightforward as it seems, as the same model can display vastly different tok/s rates depending on the testing setup.\n\nTokens are not complete words but rather chunks of text, typically around three-quarters of a character in English. The tokenizer determines how many tokens a word occupies, and this can significantly impact the cost of running the model. Models capable of producing 50 tok/s read roughly 100-150 words per second, which is much faster than a human can read. In contrast, a model stuck at 5 tok/s feels sluggish, similar to an outdated internet connection from the 1990s.\n\nThe single-user benchmark, which reports tok/s for one user and one request, does not provide a complete picture of the model's performance. When a model is deployed as an API or within a team, the relevant metric shifts to throughput - the total number of tokens generated by all users per second. The key to achieving high throughput lies in continuous batching, a technique where the server keeps the GPU busy by processing multiple requests simultaneously, rather than waiting for each request to finish before starting the next.\n\nThis concept can be likened to driving on a back street versus a highway. A quiet back street with only one car is quick, much like a single-user scenario with no traffic. On the other hand, a six-lane highway represents a high concurrency environment, where multiple requests are processed in parallel, just like a batched server. When the concurrency climbs, the batching engine significantly outperforms a single-user setup. For instance, in a benchmark conducted on an A100 40GB GPU with Llama 3.1 8B, vLLM achieved a throughput of approximately 793 tok/s when handling 256 users, compared to Ollama's 41 tok/s under similar conditions - a 19x difference that emerged only as concurrency increased.\n\nIt is crucial to understand that tok/s is not the sole determinant of a model's performance. Throughput and latency are equally important aspects that need to be considered. While tok/s describes the sustained generation capability of the model, latency refers to the time it takes for a single user to receive their response. In server environments, it is essential to pay attention to the 99th-percentile latency (P99), which represents the wait experienced by the slowest 1% of requests. This metric provides a more accurate representation of the user experience compared to average latencies, which can be misleading.\n\nAnother often overlooked metric is the time to first token (TTFT), which measures the delay between submitting a prompt and the appearance of the first word in the model's response. TTFT is primarily influenced by prompt processing, where the model reads and attends to the entire input before starting the generation process. Long context documents or multi-turn conversations can significantly increase TTFT, regardless of how fast the model generates subsequent tokens.\n\nWhen evaluating benchmarks, it is essential to consider all three metrics: tok/s, P99 latency, and TTFT. While a model may appear impressive in terms of tok/s and P99 for a single user, it may underperform in terms of TTFT for long-document workloads or other concurrent scenarios. By understanding these metrics and their implications, users can make informed decisions about which local LLM best suits their specific requirements.",
  "summary": "What tok/s really measures, how concurrency changes it, and why a single-user benchmark is not the whole story for local LLM performance. A Few Moments Later… How Fast Is \"Fast\"? Every interface in the world of local AI eventually shows you that dreaded spinner, and on the wrong setup it sits there long enough that your brain supplies the meme: \"A few moments later…\" That pause is a number…",
  "key_points": [],
  "editors_take": "Understanding tokens per second benchmarks requires considering throughput, latency, and time to first token, as a model's performance can vary greatly depending on testing setup and concurrency.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}