Multi-Provider LLM Router, or How I Got Tired of Forgetting Which API Format I Had To Use

Battling the headache of integrating multiple LLM providers like Anthropic, Google, and OpenAI can be overwhelming. Each platform presents its own Python SDK, making streaming responses via Server-Sent Events (SSE) a fragmented experience. Additionally, the formats for thinking/reasoning blocks and tool calls differ significantly.

To address this complexity, a developer has extracted the core streaming router from their platform to create an open-source FastAPI template. This solution simplifies the process into a single asynchronous endpoint: POST /v1/chat/stream. This endpoint accepts a unified request payload and returns a standardized SSE stream, emitting four clean events: thinking (internal model reasoning tokens), content (user-facing response text), tool_call (function calling requests), and done (stream completion).

The architecture relies on direct asynchronous HTTP using httpx.AsyncClient, complemented by the official SDKs for providers like Google and Anthropic. Instead of hardcoding models, the system utilizes a models.json file to dynamically maintain a catalog of providers such as Claude Sonnet 5, Gemini 3.8 Flash, and GPT 5.6 Terra. Adding new models is as simple as updating this JSON file.

The repository includes a testing playground, accessible at http://localhost:8000/, enabling users to test prompts and verify streaming latency without needing a frontend framework. Users must, however, provide their own API keys. The codebase is open-source under the MIT License on GitHub, complete with full pytest test coverage, a .env.example file, and clean Pydantic v2 schemas.