Building a coin-flip test for AI agents: 16 readings, 3 conditions, 80 Claude memos
This post walks through how a small test of AI agents was built, so that by the end you can rebuild it on your own data: invent a decision with a clear line, list the honest analysis choices in front of it, compute every reading exactly, hand the agent a coin-flipped lean, score which reading each memo stands on, and recompute every number with a second script that is designed to fail. Everything…
This article details the construction of a test for AI agents that involves analyzing a decision-making process in a fictional farming co-operative. The experiment uses Claude Code to simulate four analysis choices with two possible outcomes each, and a coin flip to determine the AI agent's leaning for each analyst. The test measures how often the AI's lean impacts the final decision, and how a brief skill training can influence its decision-making.
The experiment utilizes data from 805 closed deals across Q1 and Q2, with 16 sales reps. The AI agent's choices are compared to the actual outcomes, and the results show that the choice of which deals to count has the most significant impact on the final verdict. When only new deals are considered, all 8 readings clear the leadership threshold of 3 percentage points. However, when all closed deals are included, only 2 out of the 8 readings meet the threshold.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
