{
  "id": 2388957,
  "title": "How to Stop an Image Model From Ignoring Your Reference Image",
  "url": "https://urgent.news/2026/08/21/how-to-stop-an-image-model-from-ignoring-your-reference-image",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-21T14:16:12.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/libersand/how-to-stop-an-image-model-from-ignoring-your-reference-image-i5a"
  },
  "original_language": "en",
  "account": "A user uploaded a floor plan and attempted to remove the room labels, but the AI model failed to ignore the instruction. The model was not following the user's reference image. FloorDrafter, an AI floor plan generator, uses a reference image as a starting point and applies a requested change. The challenge lies in balancing the model's two masters: the visual reference and the user's textual instructions.\n\nSeveral lessons were learned in making the model behave correctly. First, separate the \"how to draw\" from \"what to draw\" by splitting style presets into two halves: how to render the image and what annotations to include. Second, write the rules to \"only change what is named,\" not as a list of prohibitions. Prohibitions can lock the user out, preventing them from making necessary changes. Third, explicitly authorize the requested change to avoid the model becoming overly conservative.\n\nFourth, do not ban existing content; instead, use affirmative clauses that preserve what is already present. For instance, adding a prohibition on text can inadvertently remove user-invented text. Fifth, some constraints must be immune to exemption clauses, such as geometric integrity. When a user wants to change dimensions, the model should not be allowed to duplicate or extend elements without explicit permission. Lastly, every section that can override constraints should have an exemption included. This ensures that capabilities like removing labels function as intended.",
  "summary": "A user uploaded a floor plan, typed \"remove the room labels\", and got back the same plan — labels still on it. Tried again, same result. The interesting part: the model was not ignoring the instruction. Another line in our own prompt was quietly outranking it. I work on FloorDrafter , an AI floor plan generator. A big part of the product takes a reference image — a photo of a hand sketch, a…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}