Choosing the Right Task for AI Agent Evaluation

Developers use AI agents to perform everyday tasks within your product, providing an excellent foundation for evaluating their performance. To build a comprehensive evaluation dataset, take these real-life requests and transform them into structured tasks. These tasks should account for variations developers may face and specify the information the agent must have, as well as how you'll verify the outcome.

A task dataset serves as a structured method to assess how well agents manage regular developer workflows, from tasks like installing a command line interface (CLI) to troubleshooting errors. Each task outlines the agent's goals, the data and circumstances it starts with, and the method for determining success. The following steps detail how to construct effective evaluation tasks, from selecting workflows to assessing task quality and maintaining your dataset current.

Begin by forming tasks around actual developer workflows, focusing on aspects users depend on, from initial setup to core operations and addressing frequent issues. Sources for these tasks include documentation, support requests, manual checks, and reported failures. Prioritize tasks based on their frequency of use and the impact of getting them wrong, then group them by workflow to identify gaps and monitor improvements.

Next, provide each task with a clear objective and success criteria. Define the outputs or changes that will indicate success, record the starting conditions, and allow various approaches to meet the requirements. Be sure to align the guidance with the aspect of the product you're evaluating and leave room for agents to consult relevant documentation if needed.

To further test agents' adaptability, introduce common complications to a working task, such as expired credentials, unspecified destinations, or resources that already exist. Clearly define the expected outcome for each variation, including scenarios where success might involve seeking clarification, requesting approval, or explaining why the task cannot proceed. This approach broadens the evaluation to include the decision-making process that makes the workflow practical.

Pilot tasks and verify the checks to ensure they're clear and achievable. Test the checks against known valid and invalid results, accepting different valid solutions while rejecting work that doesn't meet the requirements. Review transcripts in Quantiles to determine whether issues stem from the agent's performance or the evaluation process itself.

After finalizing the task and checks, run multiple trials to evaluate the consistency of agent performance. Reset the environment to the task's defined starting conditions before each trial, and Quantiles will automatically execute multiple trials per task without further input.

Regularly update and expand your evaluation dataset to maintain its relevance as user needs and product features change. Add new cases that broaden coverage, retain reliable workflows as regression checks, and track updates to task instructions, initial data, and validation methods. Reserve some tasks for refining documentation or skills, then use them to verify if enhancements apply to other workflows.

By following these steps, you'll develop a robust dataset that reflects real-world usage and guides your team in enhancing the user experience with AI agents.