{
  "id": 15809386,
  "title": "Respan spent two years building on LiteLLM. One upgrade made it start over.",
  "url": "https://urgent.news/2026/10/20/respan-spent-two-years-building-on-litellm-one-upgrade-made-it-start",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-20T22:13:08.000Z",
  "source": {
    "name": "The New Stack",
    "slug": "the-new-stack",
    "url": "https://thenewstack.io/respan-rust-ai-gateway/"
  },
  "original_language": "en",
  "account": "RepSpan, an AI gateway written in Rust, debuted on Tuesday after a two-year development process that resulted in over 100,000 lines of wrapper code surrounding LiteLLM. The new gateway replaces the original system with a small core and plugins responsible for managing model providers, routing, MCP tools, real-time voice, and security controls. The turning point was a LiteLLM upgrade in January 2026, which led to intermittent connection failures. Respan was unable to identify the issue after rolling back the update.\n\nRather than continuing to upgrade LiteLLM directly, the company now pulls in upstream fixes individually. To minimize maintenance headaches, RepSpan adopted an idea from Nginx, keeping the gateway core minimal and expanding functions like routing, model providers, and security through plugins. Developers can create their own plugins using the same interfaces without altering the underlying code. For example, Respan's Mistral provider only requires 22 lines of code, while handling quirks like Groq's token usage can be addressed separately.\n\nThe gateway processes incoming requests over HTTP or WebSocket through plugins in a defined order. Each plugin can determine which model handles the request, modify the response, execute extra calls, or halt the request before it reaches a provider. When both sides employ the same format, the gateway retains the original request and provider-specific fields. Furthermore, the gateway can translate between protocols, enabling applications built for OpenAI's Realtime API to connect to Gemini Live without modifications, though certain features may not be compatible.\n\nPlugins must be compiled into the gateway, which entails code maintenance and binary rebuilding. This also raises security concerns, as plugins can access sensitive data within the gateway process. While Rust helps prevent memory-safety bugs, it cannot safeguard against malicious dependencies or flawed application logic, as highlighted in a Hacker News discussion following the launch. Respan's founder acknowledged this concern, noting that developers can inspect the source code they compile into plugins.\n\nPlugin order is crucial, and Respan processes plugins in the sequence they are registered, allowing a plugin to accept or deny a request before others run. For instance, placing a cache before a security check might result in a cached response bypassing the check. Developers must consider this when configuring the gateway. RepSpan is not the only entity exploring Rust for AI gateways; LiteLLM is porting its own gateway to Rust, but the company has not yet achieved a pure Rust server with all hot paths, authentication, and rate limiting included.",
  "summary": "Respan released an open-source AI gateway written in Rust on Tuesday, after spending two years building more than 100,000 lines The post Respan spent two years building on LiteLLM. One upgrade made it start over. appeared first on The New Stack .",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}