Evaluate any agent framework with Amazon Bedrock AgentCore Evaluations

AI teams creating production agents face a challenge: as evaluation tooling has not kept up with the growing diversity of agent frameworks, evaluation systems often assume a specific SDK, LLM client, and tracing pattern. To address this issue, Amazon Bedrock AgentCore evaluations decouples evaluation from the framework choice. It works by utilizing OpenTelemetry, a vendor-neutral instrumentation framework that standardizes how distributed systems emit traces, metrics, and logs.

Every major framework supports OpenTelemetry, allowing the evaluation service to score agents regardless of the underlying SDK.

Traces, consisting of a tree of spans, represent a single step within a request and are exported over the OpenTelemetry Protocol (OTLP) to be collected by a telemetry backend, such as AWS Distro for OpenTelemetry (ADOT) which routes the spans and event records to Amazon CloudWatch. An agent's execution produces various spans, including those for model calls, tool calls, document retrieval, memory reads and writes, and orchestrations.

These spans are classified into three roles required by the evaluation service: invoke agent, inference, and execute tool spans.

The evaluation service reads these three roles from the spans and treats all other spans as contextual information. Different frameworks and instrumentation libraries may use different attribute names, nesting structures, and span-naming conventions for recording these roles, but the OpenTelemetry GenAI semantic conventions and the OpenInference specification standardize schemas for recording these three span roles.

As a result, AgentCore Evaluations can bridge different schemas and score every framework identically using the same evaluators, such as GoalSuccessRate, Correctness, Helpfulness, and custom LLM-as-a-judge.