The Same 11 Prompts, Three Copies, 48 Hours: A Free Model Regression Battery

The same prompt was read three times by the free model in one hour, resulting in three different action plans. The model's output was treated like a sensor, with only the statistics being trusted. The study aimed to determine whether the extraction code could handle a 48-hour regression run when the upstream model failed to maintain a single version of the truth.

To do this, an 11-prompt battery was created, ranging from single-key JSON requests to multi-paragraph summaries. The code then ran three copies of the model for each prompt, recording the raw output and refusing to tune anything mid-flight. The diversity ratio was used to measure the stability of the output, with values near 0.33 indicating agreement, values near 1.0 indicating noise, and everything in between being too small a sample to interpret.

The study found that short prompts were stable for a while but then flipped an entire key name, while long prompts showed high diversity from the first run and never improved. The flakiest part was not the model, but the assertion layer, as the output contract was defined too narrowly. The diversity ratio was only useful at the extremes, and a score of 0.33 meant agreement, while a score of 1.0 meant noise.

The study concluded that freezing the prompt list fiercely, keeping three cheap assertions, storing raw text and hashing it to JSONL, and dropping any prompt that showed 1.0 twice in a row were crucial steps to ensure stability. The study was performed using MonkeyCode's free model access and hosted on its free server option, as the author did not want to keep their laptop awake for two nights.