How monday.com Runs Agent Evals Against Real Dependencies: Webinar Recap
An agent eval suite's outcome can only be trustworthy if it's operating in an environment similar to production. You can have the best grading logic in the world, but if the agent is calling mocked databases and fake APIs, you're not testing how it behaves in the real world, you're testing how it behaves in the mocks you built. That gap is what we dug into on September 15, when we sat down with…
Monday.com recently held a webinar demonstrating how their team implemented agent evaluations against real staging clusters, rather than using mocks. Agent evaluations check the entire process an agent takes to arrive at an answer, not just the final output. This is crucial because agents are non-deterministic and can choose different tools or paths on each run.
However, mocks have limitations - they drift from production environments, the data is unrealistic and lacks edge cases, and they cannot represent state changes. Staging environments, on the other hand, are current with production, run on real data, and allow checking the agent's real end state. The main challenge with staging is that it's not lightweight or isolated like mocks.
To address this, Monday.com created Mirrord, a tool that connects local processes to a real Kubernetes cluster, mirroring real traffic and injecting real environment variables and secrets. This allows agent evals to run in minutes instead of hours, and costs close to zero as they already pay for staging. Using Mirrord, Monday.com was able to catch regressions in their model upgrade, where goal completion dropped by about 10% and agent correctness dropped by a similar amount.
The suite failed to meet the pass threshold, alerting the team to specific issues with tool selection and prompt adherence, guiding them towards a fix before shipping.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
