Per-Branch AI Endpoints: Isolating Model Spend Across Prod, Preview, and CI

Neon, a cloud platform, has introduced a solution to make AI spend more visible and manageable by isolating model spend across different environments such as production, previews, and CI (continuous integration) branches. Previously, all AI spend was attributed to a single gateway credential used across these environments, making it difficult to track costs, set caps, or receive early warnings about excessive spending on non-production environments.

The new approach allows each branch in Neon to act as its own deployment, with its own function endpoint and usage ledger stored in Postgres. This means that calls made on a branch are logged and recorded against that specific branch’s ledger, providing a clear and separate record of usage for each environment. When a preview or CI job is run, it only impacts its own ledger and does not affect the production ledger.

To demonstrate this, a test was conducted where two model calls were made on a CI branch, which increased the token count on that branch but left production unchanged. This shows that spend is now isolated to the branch it occurred in, preventing one environment from depleting the budget or rate limit intended for production.

The isolation of spend is achieved through copy-on-write technology, where a new branch inherits the ledger snapshot of production at the time of branching. Any new spend on the branch is recorded separately and does not affect production. This method not only provides clear attribution for each environment's spend but also enables containment, where a runaway job or load test on a preview branch cannot affect production's budget or rate limits.

By moving the usage ledger onto the branch, developers can easily query their environment's spend by grouping tokens by model from the branch’s usage log. This enables straightforward tracking of costs for each environment, such as "what did this preview cost?" The solution also simplifies cleanup, as deleting a branch automatically removes its ledger, eliminating the need for separate pruning of outdated data.

In summary, Neon's new per-branch AI endpoints provide a transparent and manageable way to track AI spend across different environments. This setup allows for detailed attribution, containment of costs, and easy cleanup, making it easier for organizations to budget and monitor their AI usage effectively.