The Cost of Self-Hosting Open-Source LLMs: GPUs, Staffing, and Operations

Self-hosting a large language model (LLM) in production can cost anywhere from $150,000 to over $2 million per year in 2026, depending on the size of the deployment. While GPUs are a major part of the expense, the bulk of the costs come from staffing and operations.

For small deployments with 7B-32B models on 1-2 rented GPUs, the annual cost ranges from $150K to $300K. Most of this is staff time, with 1-2 engineers handling maintenance and updates, costing around $100K-$200K.

Mid-size deployments with 70B-class models on 4-8 rented H100 or H200 GPUs cost $400K-$900K annually. Staffing this setup requires 2-3 engineers, totaling $250K-$500K per year.

Larger deployments at $1.2M-$3M per year involve 16-32 H200 or B200 GPUs spread across regions. Here, 4-6 engineers are needed, with fully loaded costs around $600K-$1.2M.

Running the necessary serving stack, autoscaling, monitoring, security reviews, and model upgrades adds further expenses, regardless of usage. The most common mistake is overlooking these hidden costs.

When deciding GPU needs, memory capacity and expected traffic are key factors. For 7B-14B models, a single 24-48GB card suffices, while 32B models may need one H100 or two smaller cards. 70B models typically require an 80GB H100, and larger models like DeepSeek-class or gpt-oss-120B demand a full 8-GPU node, with double that for production use.

The cost per token is heavily influenced by GPU utilization. Using SemiAnalysis InferenceX benchmarks, one H100 can serve about 1,465 tokens per second at interactive speeds, costing roughly $0.25 per million tokens when fully utilized. At 20% average utilization, this cost jumps to $1.25 per million tokens, highlighting the impact of underutilized resources on profitability.

Staffing costs often represent the largest expense for self-hosted LLMs, as production demands a dedicated team of engineers to manage GPUs, serving software, and on-call support. Roles include ML infrastructure engineers, MLOps engineers, and optional applied ML engineers, with salaries ranging from $130K-$220K. Adding benefits, taxes, equipment, and recruiting costs inflates these figures significantly.

With only one to three engineers typically available for on-call rotations, the risk of downtime is high, making staffing a critical cost factor in self-hosting LLMs.