Compare Against the Schema They Shipped, Not the One You Expected
My harness flagged the model for sending the wrong arguments. It compared what the model actually sent against what the run had committed to in advance, and they did not match. The mismatch was real. The conclusion I drew from it was wrong, and the comparator could not have told me so. Here is the check that was incomplete, the check that replaced it, why the fix is not "loosen the comparison,"…
The harness compared the model's actual arguments with the expected ones, and they did not match. The mismatch was real, and the comparator couldn't tell that the conclusion was wrong. The check that was incomplete was replaced, and the fix is not to loosen the comparison. The setup involved the harness preparing an exec call before the model runs, freezing it, and then comparing the model's actual tool arguments against the frozen one.
If they differed, the run failed closed. The expectation was built with specific elements, but there was a key error in understanding the provider's exec schema and the implementation schema wired into the sandbox exec tool. The harness's instruction asked for an exact JSON argument object, but the model followed the provider's schema instead, leading to a mismatch.
The fix corrected the expectation and made the comparison gate stricter, ensuring that only the required key and intent value were used.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
