Gemini reads the floor plan. Code decides what it means.
I created this post for the purposes of entering the All Things Agentic Hackathon. My wife and I have looked at twenty-two houses this year. We still haven't bought one. It isn't that nothing's available. It's that what rules a house out for us isn't on Zillow. You can filter beds, baths and price. You can't filter which way the front door faces - and for my family, Vastu isn't negotiable. You…
This post was written for the All Things Agentic Hackathon. The author's wife and she have toured 22 houses in the past year, yet none have been purchased. The issue isn't the availability of houses; it's that certain criteria for their suitability aren't present on Zillow. These criteria include the direction of the front door, the presence of a full bathroom on a downstairs bedroom, whether the backyard is flat enough for a toddler, and the proximity of the lot to a four-lane road.
The author discovered that these factors are not listed as fields in any listing feed; they are instead contained within the floor plan and satellite photo. Thus, the problem isn't a search issue, but a seeing problem.
The author initially built a system that appeared to analyze floor plans. This system was able to provide scores, findings, and confident write-ups about the entrance orientation and yard grade. However, upon closer inspection, it was discovered that the system was only working from the street address alone, not receiving any floor plans.
This led to slow performance (30 seconds per scan) and inaccurate results (3 out of 5 aerial images being unavailable). The solution was to separate the components of the system. Gemini was designated for perception (visual input and analysis), while code was responsible for judgment (scoring and decision-making).
This separation resulted in several benefits. First, results could be cached, allowing for quicker retrieval of information about a house. Second, scores became reproducible, ensuring that the same house would receive the same score every time. Third, it became possible to swap rulebooks (e.g., Vastu and Feng Shui) without needing to call Gemini. Finally, accuracy could be measured, as opposed to measuring a model against scraped data, which would have resulted in no answer key.
The author generated floor plans using Gemini 3.1 Flash Image, written based on specifications. This approach provided a definitive answer key, allowing for the measurement of the model's accuracy. The system runs offline in about a second, enabling anyone to rerun the process. The three adjacent misses were due to the model stating North where the specification said North-East, which is a legitimate disagreement between two surveyors.
The author initially assumed the Gemini-3.5-flash was the wrong default. However, they later discovered that Gemini-3.5-flash-lite provided the same accuracy in 1.5 seconds, satisfying the hackathon's model requirement. The solution also included a fallback chain and recording which model produced each reading. When the model failed (503 UNAVAILABLE), the issue was due to capacity on Google's side, not a rate limit.
The Firestore client was also found to throw an uncaught exception when credentials were missing, causing the process to die. The solution was to check for credentials before constructing the client. Lastly, the feedback feature did not work as expected. It adjusted global weights, but a score is a weighted average over seven dimensions.
Raising a weight for a dimension a house scores well on would only increase the score by less than a point, rendering the feedback feature ineffective.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
