The model obeys your schema, not your description

Two models were asked to follow the same prompt and tool description to generate a response. One model returned a pattern with an entity name and definition fields, while the other returned a pattern with just the entity name and fields. The second model's response was rejected by the downstream patcher, causing confusion and wasted time.

The issue was traced back to the exposure gap between the models - the "smarter" model had seen the shape before and was able to retrieve the necessary memory, while the less trained model guessed based on the description text and produced an incorrect result. The root cause was the lack of a strict schema for the tool, which led to the model guessing the pattern instead of following the prescribed format.

To address this issue, the team focused on writing tight schemas for the common pattern kinds, while providing a more permissive fallback for the rare ones. This approach targeted the failures they actually encountered, rather than investing time in creating schemas for every possible shape.