If you don't run your own LLM proxy, how do you know whats going on?

The title of this article captures the core idea: if you're using a third-party proxy for a large language model (LLM), you have no way of knowing what's really happening behind the scenes. You might be using a software as a service (SaaS) application that automates various tasks, such as cloning Labubu release images or searching for macronutrient information for a McRib.

You're likely interacting with a frontend that connects to a paid subscription service. The provider shows you a proof of work, but that's not useful for engineers trying to make informed decisions.

The author argues that seeing token usage statistics with time series data is essential for understanding what's going on. The analytics dashboard provided by the provider is a distraction, not a valuable tool. They can't make decisions based on this information, as it's not insightful enough. Some providers offer OpenTelemetry (OTel) export options, but the author emphasizes that running the proxy yourself is the best approach.

They suggest modifying the proxy to fit your specific needs, such as custom MCP server endpoints or additional features.

The author provides examples of how running a proxy can help with troubleshooting, like pinpointing cache misses during coding sessions. They also mention the possibility of automating the proxy with an API. Owning your infrastructure, application visibility, data pathing, and storage is crucial. The author does this at home for personal learning and at work to ensure they're doing their job properly. While they may not fully understand the intricacies of LLMs, they can still effectively monitor and manage the system.