Urgent.News

What's breaking now, across thousands of outlets.

More in AI

The Cost of Self-Hosting Open-Source LLMs: GPUs, Staffing, and Operations

Self-hosting an open-source LLM in production costs roughly $150,000 to $2 million or more per year in 2026. GPUs are only part of the bill: for most companies, the engineers who run the stack cost as…

  • Self-hosting LLMs costs $150K-$2M annually, depending on size and GPU count.
  • Staffing and operations dominate costs, with 1-6 engineers needed for maintenance.
  • Hidden expenses like serving stack, monitoring, and upgrades add significant costs.

Structured Output Doesn't Validate LLMs. It Resamples Them

TL;DR — Constrained decoding for structured output (JSON mode, grammar-based function calling) doesn't just check a model's answer — it rewrites the model's token-by-token probability distribution in…

  • Structured output reshapes model's token distribution in real-time.
  • Schema design intervenes in sampling process, not validates model.
  • Grammar constraints force low-probability content, fine-tuned JSON degrades gracefully.

More from Friday 6 November →