The AI observability race will be won on interoperability

The AI race is increasingly judged on model quality, speed, and token costs. However, businesses must consider if they can reliably deliver useful outcomes at the right cost while managing associated risks. To achieve this, organizations require comparable evidence and data across various models, frameworks, and providers. OpenTelemetry provides a common method to capture and describe AI behavior, reducing the need for separate data models for each integration.

Adopting shared AI conventions can lower duplicate integration work, facilitate easier comparisons between providers and models, and aid decisions on quality, cost, and risk. Consistent data also strengthens the basis for scaling AI applications. The AI landscape comprises numerous components such as model providers, orchestration frameworks, retrieval systems, tools, and evaluation services.

Each may handle conversational or agent actions differently, leading to inconsistencies that can hinder failure investigation and result comparison. Standardization does not necessitate identical data storage across all systems but requires a shared understanding of critical aspects. OpenTelemetry's GenAI conventions outline common conversation and message concepts while acknowledging the limitations of representing complex data in telemetry.

They also enable the storage of extensive content externally, with references retained in telemetry. Splunk's Agent Observability work demonstrates the practical value of this approach by offering vendor-neutral monitoring across AI agents and applications using OpenTelemetry and AGNTCY standards. The goal is to enable teams to correlate agent behavior with performance, quality, cost, and risk, even as the underlying stack evolves.

Token counts assist in understanding expenses, but they do not indicate whether AI spend resulted in useful work or added business value. A cost-effective model might consume more tokens or need more retries to complete a task. Conversely, a pricier model could prove more efficient if it achieves the desired result reliably on the first attempt.

Therefore, cost per outcome is a more valuable business metric than cost per token alone. A successful outcome could be a completed workflow or an accurate response meeting predefined quality standards. OpenTelemetry facilitates the collection of consistent data regarding model calls, latency, errors, usage, and tool interactions, irrespective of the model, backend, provider, or tools employed.

However, organizations must still define success metrics and measure them against the cost and utilization of these tools. Token counts provide a means to explain expenses, but they do not reveal whether AI spend led to useful outcomes or enhanced business value. A less expensive model may consume more tokens or necessitate more retries to accomplish a task.

In contrast, a more expensive model might be more efficient if it accurately completes tasks on the first try. Consequently, cost per outcome proves a more beneficial business measure than cost per token alone. Evaluation can aid this process. Instrumentation-side evaluators offer real-time feedback close to the application, while platform-side evaluations can compare models and releases across workloads.

With consistent telemetry, teams can leverage evaluation results to determine the most suitable model for a task, assess whether a release improved performance, and identify areas where spending generates value. They can perform these evaluations regardless of changes to models, providers, or tool calls. To ensure change management without the need for reinstrumentation, telemetry quality must be evident before production.

Open source tools like Observability Studio provide a development-time example, enabling teams to inspect local telemetry and verify adherence to OpenTelemetry conventions. They can even leverage the tool to facilitate compliant instrumentation from the outset. Identifying missing or inconsistent data early can minimize rework and equip operators with stronger evidence when troubleshooting or assessing releases.

OpenTelemetry's framework for collecting consistent data, regardless of the model, backend, provider, or tools used, enhances the business benefit by expediting the transition from adopting a new AI capability to understanding its behavior in relation to the organization's business priorities and customers. This control over AI agents is crucial since they can select tools, retry operations, and execute actions on behalf of users.

An AI SRE agent, for instance, may detect a customer checkout issue on a retail site, investigate the problem, restart a failing service, and confirm that purchases are functioning correctly. Teams must determine not only whether the agent responds but also if it takes meaningful actions, adheres to safety rules, and knows when to cease and seek assistance.

A misguided action can exacerbate outages and harm customer trust, while excessive retries or endless troubleshooting can result in unexpected costs. Observability can provide operational control in this scenario. Agent observability enables teams to evaluate behavior and monitor performance and cost. When integrated with enforcement systems, it can also support guardrails that prevent harmful actions or outputs.

Shared telemetry, collected through OpenTelemetry, connects these controls to evidence regarding the agent's actions, tool calls, and ultimately, its business impact. The standard should make evidence portable; however, the business defines the policy for applying that evidence. Teams must decide which actions require approval, what quality bar is acceptable, and when a request should be routed to a person or a different model. Upstreaming changes to open source can provide broader coverage and benefit the entire ecosystem.