A 10-week roadmap to run LLMs in production

This 10-week roadmap equips engineers to deploy and optimize large language model (LLM) inference services in production. Over fifty sessions, participants build an OpenAI-compatible inference service running on rented GPUs, instrumenting, load testing, quantizing, and optimizing for concurrency, memory usage, and cost-aware routing. The alternative approach of multiple disconnected experiments wastes time on setup.

Comfort with Python, transformers, and command line is required. No prior serving, Kubernetes, or CUDA experience is needed. A single service grows with coverage, eliminating the need to show multiple experiments. Participants should be comfortable with the content, skipping skim sessions. Buffer days allow for missed sessions without collapsing the plan.

The roadmap's order emphasizes five key decisions: the roofline model, quantization, continuous batching, speculative decoding, and disaggregation. Measurement precedes optimization. Instrumentation occurs in week 3, followed by load testing in week 5. These measurements are essential for verifying subsequent optimizations and creating a public benchmark.

Key concepts like paged attention and continuous batching are not standalone build exercises; they are implemented in vLLM, which participants study. The router, which chooses backend execution based on cost, latency, and quality, introduces unit economics into the service design. Edge deployment is optional and occurs at the end.

The first five weeks focus on building the inference service without a GPU. Week 1 builds foundational concepts, while weeks 3, 6, and 9 require no GPU usage. Week 5 involves testing 1000 concurrent requests, and week 8 covers disaggregation if applicable. Docker, kind or a small managed cluster, and an H100 GPU are used for weeks 1, 3, 6, and 9.

By the end of week 9, participants have a hand-derived arithmetic intensity figure for the model being served and the ability to explain why specific phases are bandwidth-bound. The service becomes OpenAI-compatible, serving a 7-8B model, with accompanying notes explaining essential concepts from the source material.