Use decision models with Amazon Bedrock AgentCore Evaluations
Some agent-evaluation questions are narrow and repeatable, and decision models answer exactly that kind of question. This post shows how to connect two decision models, Jev and Strands Decider, to Amazon Bedrock AgentCore Evaluations through code-based evaluators, and how to validate a decision-model check before you gate a release on it.
Evaluating an agent using decision models can provide a clear label and a number for scoring purposes. These models take a structured state and a fixed set of allowed answers, returning a calibrated probability over them. Amazon Bedrock AgentCore Evaluations supports custom evaluators, which can be either LLM-as-a-judge or code-based evaluators.
Custom LLM-as-a-judge evaluators apply instructions and rating scales with a model in Amazon Bedrock, while code-based evaluators run logic in an AWS Lambda function. Decision models are suitable for questions with a fixed set of answers that can be decided quickly, such as whether an agent refused a request or remained on topic.
They are also useful for enforcing a specific rubric consistently. This post demonstrates how to connect two decision model backends to AgentCore Evaluations through code-based evaluators, specifically Jev and Strands Decider 2B.
Written by urgent.news from AWS Machine Learning's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
