How to Stop an Image Model From Ignoring Your Reference Image

A user uploaded a floor plan and attempted to remove the room labels, but the AI model failed to ignore the instruction. The model was not following the user's reference image. FloorDrafter, an AI floor plan generator, uses a reference image as a starting point and applies a requested change. The challenge lies in balancing the model's two masters: the visual reference and the user's textual instructions.

Several lessons were learned in making the model behave correctly. First, separate the "how to draw" from "what to draw" by splitting style presets into two halves: how to render the image and what annotations to include. Second, write the rules to "only change what is named," not as a list of prohibitions. Prohibitions can lock the user out, preventing them from making necessary changes. Third, explicitly authorize the requested change to avoid the model becoming overly conservative.

Fourth, do not ban existing content; instead, use affirmative clauses that preserve what is already present. For instance, adding a prohibition on text can inadvertently remove user-invented text. Fifth, some constraints must be immune to exemption clauses, such as geometric integrity. When a user wants to change dimensions, the model should not be allowed to duplicate or extend elements without explicit permission.

Lastly, every section that can override constraints should have an exemption included. This ensures that capabilities like removing labels function as intended.