I built an open-source AI gateway for Kubernetes: budgets, kill switch and evals in CI
Every team wants AI. Then three questions arrive: What does it cost? Where are the keys? Can we stop it? I'm a platform engineer, and I built SanVi to answer all three on Kubernetes. The gateway Apps call one OpenAI-compatible endpoint (LiteLLM engine) with their own virtual key. Each key has its own models, budget and rate limit. No key → 401. Over the rate limit → 429. Over budget → blocked.…
A platform engineer named Sandhilk has created an open-source AI gateway called SanVi for use with Kubernetes. This gateway aims to address three common questions about using AI: what it costs, where the keys are, and how to stop it. The gateway functions as an AI access point, allowing apps to call a single OpenAI-compatible endpoint that utilizes a LiteLLM engine.
Each app has its own virtual key, which is used to manage models, budgets, and rate limits. If an app attempts to use a key that doesn't exist, it receives a 401 error. Should the app reach its rate limit, it will receive a 429 error. If an app exceeds its budget, the gateway blocks its access to the AI service. All calls made through the gateway are logged, including details about the team, model, tokens used, and the associated costs, all stored in a Postgres database.
The gateway's keys are never hard-coded into the system; instead, they are securely stored in a Vault, with the keys being synced through External Secrets. Sandhilk has implemented a kill switch that can be activated in the event of a leaked key, runaway agent loop, or a sudden spike in spending. By executing the block function in the kill_switch.sh script, the gateway can instantly prevent further access to the app's team, returning a 401 error.
Once the issue is resolved, unblocking the team will restore access. The system is designed to allow developers to manage AI access as code, with teams, models, and budgets defined in a file named access/teams.yaml. Any changes made to this file are submitted through a pull request, and the CI system checks the policy of the changes before applying them via Argo CD.
The Right-Size Advisor is a built-in feature that runs an evaluation suite on each model to determine if a more cost-effective alternative exists. In the case of Sandhilk's workload, the Haiku model proved to be 95% cheaper and 40% faster than the previous Sonnet model. For any change that results in a drop below 80% efficiency, the CI system will fail.
Sandhilk emphasizes that the deployment agent can transform a single sentence into a validated Dockerfile and Helm chart, but it can only open a pull request. Additionally, the system is designed to be audit-ready, with a compliance script checking twelve controls (including no secrets in Git, least-privilege access, an audit trail, a kill switch, and a quality gate) against standards such as ISO 27001, SOC 2, and the EU AI Act.
These checks are performed on every change. To try SanVi out for yourself, the source code, quickstart guide, and Helm chart are available on GitHub at https://github.com/sandhilk/sanvi-gateway. Sandhilk invites blunt feedback and is interested in learning about any additional requirements developers might have before implementing this gateway in their own companies.
Rewritten by urgent.news; may contain errors — check the original.