I built CI for prompts, and the first bug was in the tests

Sentinel is a tool that guards against prompt regressions in code changes. It evaluates a prompt's performance before and after modifications, allowing it to halt the merge process if significant issues are detected. Developed quickly for a hackathon, Sentinel's source code can be found at https://github.com/ParthGupta1304/sentinal.

One of Sentinel's primary challenges lies in its inability to verify test inputs. The judge it interacts with only pays attention to the output, not the input, making rubrics that reference "the input" unanswerable. This oversight led to the judge's honest assessment of the issue, resulting in a lower score. To address this, the team decided to pass the case input, allowing for a more accurate evaluation.

Another issue arose when the temperature parameter was rejected by Claude-sonnet-5, causing the first calibration pass to fail every rubric case with an HTTP 400 error. Stability was achieved by disabling thinking, rather than by setting temperature to 0. Four criteria were set, but one of them contradicted the prompt's documented behavior, leading to the deletion of the rubric.

The subject of the test, rather than the judge, often changed during the evaluation process. GPT-4o-mini returned an empty improvements array approximately one-third of the time, without any prompt changes, resulting in a noise floor of 0.80 on a 0-1 scale. This made it nearly impossible to detect any dropped regressions. To resolve this, the rubric was deleted, as it could never produce a meaningful verdict.

An early instruction stated "do not merge," but the model dutifully refused to merge, never triggering the tool.approval_required event. The human never saw a checkpoint, as the instructions were updated to require explicit merging commands. The demo regression occurred when a prompt was tightened by removing the line stating that if there is nothing to evaluate, the score should be 0 with a fixed sentence.

While the JSON would still parse in production, Sentinel's strict validation caused the entire test to fail. In one re-run, the new prompt scored the empty submission 16, highlighting the critical importance of the initial prompt's integrity.