Why I Didn't Put a Proxy Between My App and OpenAI

Every AI cost attribution tool works by intercepting every request. You point your base URL at their gateway, they monitor each request, and in return you receive real-time alerts if a call threatens to exceed your budget. This trade-off is acceptable for many teams, but the author had a different priority. The most important consideration for them was uptime.

Routing through a gateway introduces a new dependency and network hop, which affects reliability. For teams focused solely on understanding feature costs without interrupting functionality, this added complexity was undesirable. Consequently, the author developed a wrapper instead of a proxy. This wrapper directly integrates with the OpenAI and Anthropic Python SDKs, requiring no changes to your existing code.

The wrapper measures call duration and logs costs asynchronously after the response is received. The core challenge was ensuring that a malfunctioning reporting endpoint would not disrupt the primary API call. The author tested this by simulating a dead ingestion API, verifying that the actual request still succeeded, even in the absence of reporting.

This approach provides per-call cost data with customizable tags for features, teams, or users. However, it does not halt calls before they occur, which is a feature of gateways like LiteLLM or Portkey. The tool is particularly beneficial when visibility on costs is paramount, and blocking is not a concern. Currently, streaming responses are not supported, so it's best suited for non-streaming calls.

The wrapper is MIT licensed and in early development (v0.1.x), with PyPI installation available via pip install cognocient. The author invites feedback on whether this solution addresses cost visibility concerns without adding unnecessary dependencies.