From Prototype to Production: Deploying an LLM App That Won't Collapse

A founder of an e-commerce startup reached out for help when their search and summary assistant, built to run on a laptop, began to struggle under load. With 12,000 products in the catalog, the demo was impressive, but load testing with 40 simulated users caused response times to skyrocket. The memory usage on the model server also increased to 30 GB, and HTTP 504 errors started appearing. The founder asked if adding more servers would solve the problem, but the truth was that this approach was not the solution.

The issue was not merely hardware-related; it was an architectural one. The e-commerce team had built a prototype that failed to scale and handle real traffic. To understand the difference between a prototype and an application, one must ask what happens when 40 users use the service simultaneously. A prototype would crash, while an application would respond with a queue, rate limit, fallback, or scaled-up worker.

Four key properties distinguish a prototype from an application: statelessness, clear boundaries between layers, backpressure, and observability. Statelessness means that the service must be restartable at any moment. Boundaries between layers, such as model serving, API logic, and state storage, should be separate processes with explicit contracts.

Backpressure ensures that when demand exceeds capacity, the system degrades gracefully instead of silently dropping requests. Lastly, observability is essential to be able to measure and fix issues in the system.

The e-commerce team had none of these four properties in place. Their model, FastAPI app, and session cache were all part of the same process, leading to blocking requests and memory growth when the inference slowed down. The solution was not to add more servers but to change the architecture.

Three main steps were identified to create a production-ready application: choosing how to serve the model, the architecture that survives, and a production-shaped API. The choice of serving the model depends on the cost per 1 million tokens. Hosted APIs are more expensive but have near-zero operational load, while self-hosted GPU solutions offer a balance between cost and performance. Quantized CPU models are also an option, with surprisingly robust performance.

The recommended approach is to use a hosted endpoint for the main model and self-hosted only for high-volume or data-sensitive workloads. If there are constraints on data placement, start with a hosted endpoint and only move to self-hosted infrastructure when the need arises. Regardless of the choice, put the model behind a single interface so that it can be swapped easily with an environment variable.

The architecture for a production-ready application should include a client layer, an Nginx server for TLS and rate limiting, API replicas running FastAPI, and Redis for sessions, cache, and queue, as well as Postgres for facts and audit logs. The model should be behind the same contract, whether it is a local vLLM server or a hosted endpoint. Finally, the API layer should be stateless, with an explicit input budget and a hard timeout on the model call.

The FastAPI code provided demonstrates the shape of the production API, including authentication, loading the last 10 turns from Redis, calling the model with a hard timeout, and persisting the turns. The asyncio.wait_for line is crucial to prevent slow models from backing up every worker and causing the queue to grow.