{
  "id": 15807846,
  "title": "Coordinate Clicks Are a Trap: Driving a GUI Agent Without Pixel Guessing",
  "url": "https://urgent.news/2026/10/20/coordinate-clicks-are-a-trap-driving-a-gui-agent-without-pixel",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-20T22:03:56.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/galagent/coordinate-clicks-are-a-trap-driving-a-gui-agent-without-pixel-guessing-452i"
  },
  "original_language": "en",
  "account": "The most time-consuming debugging challenge I encountered while building my GUI agent was dealing with stale screens. Unlike ambiguous no-ops that could either indicate a failed action or a silent failure, stale screens presented a unique problem. Screenshots become outdated within seconds, and caching the plan rather than the pixels is crucial. The agent must constantly verify if the screen has changed, if the expected text appears, or if the node count has altered, all with just a single boolean per step. Failure to do so results in the agent performing actions on outdated information, leading to incorrect outcomes. This issue is particularly critical when dealing with rapidly changing user interfaces, where even a brief delay can lead to significant errors.",
  "summary": "The first version of my GUI agent clicked by pixel coordinates. It worked in demos and failed constantly in practice, and the reason is boring: pixels do not move, but content does. Here is what I changed, in order of impact. 1. Read the tree before you click Both desktop and mobile expose an accessibility tree: uiautomator dump on Android, UI Automation on Windows, the accessibility APIs on…",
  "key_points": [
    "Stale screens become outdated within seconds",
    "Agent must verify screen changes, text, node count",
    "Failure leads to actions on incorrect information"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}