The agent evaluation playbook: Built-in, DeepEval, and AutoEval on Amazon Bedrock AgentCore Evaluations

Deploying agents in production requires more than just checking if they function. It's about ensuring they meet business requirements at scale. Teams often rely on surface-level signals, such as latency and user feedback, which do not necessarily indicate whether the agent successfully completed the user's task or adhered to privacy rules. Evaluating agents properly is crucial, as a high completion rate may hide issues like leaking personal information.

To achieve accurate evaluation, it's essential to measure the behaviors that matter for each agent, at the appropriate level of detail. Amazon Bedrock AgentCore offers built-in evaluators, as well as DeepEval and AutoEval evaluators, to help with this process. Instead of starting with metric names, define success in business terms and break it down into observable behaviors.

For example, a customer support agent aiming for first-contact resolution should complete the user's goal, stay coherent, maintain compliance, and invoke the correct backend tools.

AgentCore's evaluator catalog consists of three tiers: Built-in evaluators, third-party managed evaluators from DeepEval and AutoEval, and custom evaluators. Built-in evaluators are quality-tested by AWS, offering a reliable starting point. Third-party managed evaluators provide control over the model and account used for inference. Custom evaluators allow for more flexibility by enabling the use of your own models and modifying prompts or rubrics.

The evaluator landscape includes various groups such as Response Quality, Task and Goal Completion, Tool Use, Skills, Safety, and Trajectory Checks. Built-in AWS evaluators cover response quality dimensions like correctness, faithfulness, helpfulness, relevance, conciseness, coherence, instruction following, and refusal. Task and goal completion is measured using GoalSuccessRate, while Tool Use assesses the agent's selection accuracy and parameter accuracy.

Skills evaluators determine if the correct skill was invoked and followed its steps. Safety evaluators focus on harmlessness and stereotyping, while trajectory checks compare the agent's actual tool path against the expected one.