AWS Bedrock's built-in eval graded my agent green. It only looks at a quarter of it.

I built an agent on AWS Bedrock using a low-code approach. First, I selected a foundation model, then I set up an action group linked to a Lambda, attached a Knowledge Base, and added Guardrails. After that, I ran Bedrock's built-in Model Evaluation job and obtained good scores, subsequently shipping the agent to production. However, production disagreed almost immediately, as the agent was dropping a significant portion of its tool calls, which the built-in evaluation missed.

This issue stemmed from the fact that Bedrock's built-in evaluation only assessed the model itself, while the agent comprises four components – the model, action groups (tool-call layer), Knowledge Base (retrieval layer), and Guardrails (safety layer). The built-in evaluation did not account for these additional layers, leading to a quarter of the user experience being ungraded.

To address this, I identified three key aspects to evaluate: the accuracy of tool calls (action groups), the relevance of retrieved documents (Knowledge Base), and the effectiveness of Guardrails in detecting attacks without obstructing legitimate user queries. Initially, the built-in evaluation provided a one-time assessment, with no mechanism for re-evaluating after changes.

Thus, I implemented periodic evaluations for each component, rather than a single assessment at the start of deployment. This revamped approach enabled me to pinpoint and resolve issues more efficiently.

It's crucial to understand that Bedrock's model swap is a one-line change, but it may result in a different performance across models. Hence, running the same evaluation suite on every model the agent could potentially resolve to is essential. This includes testing on the original model, as well as any subsequent replacements. Additionally, I learned that non-English content can sometimes be overlooked during evaluation, leading to unnoticed performance degradations in specific languages.

By carefully monitoring retrieval quality for each language separately, these issues can be avoided. When evaluating Bedrock agents, it's vital to assess all components independently, rather than relying on a single aggregate score. This way, you can identify the precise areas that require improvement and make targeted adjustments to enhance the overall performance of your agent.