TokenSpeed: A New LLM Inference Engine Built for Coding Agents.
If you self-host large language models, you have probably settled into one of two camps. You use vLLM or SGLang because they are easy to run, or you use TensorRT-LLM because it squeezes the most out of NVIDIA hardware and you are willing to pay for that in setup pain. TokenSpeed wants to end that trade-off. It is an open-source (MIT) inference engine from the LightSeek Foundation that targets…
TokenSpeed is an open-source inference engine developed by the LightSeek Foundation that aims to achieve TensorRT-LLM-level performance with vLLM-level usability. It is specifically designed for agentic workloads, such as coding assistants. Most inference benchmarks focus on short prompts and answers, but coding agents require very long contexts, many turns, and fast generation times. TokenSpeed targets around 70 tokens per second per user as a minimum, sometimes 200+.
The engine consists of four main components: the modeling layer, scheduler, kernels, and the entrypoint. The modeling layer handles tensor-parallel communication code generation based on module boundaries, which reduces bugs when adding new models. The scheduler, written in C++, models the request lifecycle and KV cache ownership using a finite-state machine, ensuring safe KV cache reuse.
Kernels are pluggable packages with a registry, allowing NVIDIA and AMD backends to coexist. The entrypoint is an OpenAI-compatible server that minimizes CPU-side overhead.
To get started, users can use a preinstalled Docker image or install via pip with nightly wheels. The recommended method is Docker, which pulls the runner image and starts the server with specified parameters. Once the server is running, users can call it like OpenAI, using the same client code with a base_url change. The team has reported performance improvements over TensorRT-LLM, achieving about 9% faster latency at batch size 1 and 11% more throughput at 100 tokens per second per user.
TokenSpeed's MLA decode kernel also cuts latency roughly in half for typical speculative-decoding workloads.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
