{
  "id": 14382193,
  "title": "Your computer-use agent clicks 20 pixels off: screenshots vs the accessibility tree",
  "url": "https://urgent.news/2026/10/14/your-computer-use-agent-clicks-20-pixels-off-screenshots-vs-the",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-14T06:04:35.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/autenai/your-computer-use-agent-clicks-20-pixels-off-screenshots-vs-the-accessibility-tree-4dcg"
  },
  "original_language": "en",
  "account": "When a computer-use agent needs to click an element on the screen, it must first determine what is currently displayed. There are two ways to accomplish this: send the model a screenshot to analyze the pixel grid, or query the operating system's accessibility tree, which already contains the role, name, state, and position of each control. Most demos use screenshots because they work universally, but a combination of both methods yields better results.\n\nThe accessibility tree provides several advantages over a screenshot. It clicks the actual element rather than a guessed coordinate, informs the agent of the button's state (e.g., disabled, checked, focused), and eliminates the need for the model to infer details like a greyed-out button from a slightly lighter shade. Moreover, a tree is cheap to generate; a screen with 40 controls typically requires only a few dozen short lines of text, while a screenshot costs roughly width x height / 750 tokens per step.\n\nHowever, the accessibility tree can be incomplete or misleading. Canvas apps, design tools, remote screens, Electron and Chromium applications, and custom-drawn controls with no names can all produce an empty or inaccurate tree. Similarly, stale positions during animations and scrolling can cause issues. In these cases, screenshots remain a useful backup option.\n\nTo ensure reliable performance, it is essential to understand how each approach works and where they might fail. The accessibility tree can be inspected using tools like the Accessibility Inspector on macOS, Accessibility Insights for Windows, or Accerciser on Linux. For Android devices, adb shell uiautomator dump can be used to generate the view hierarchy.\n\nThe recommended workflow is to prioritize the accessibility tree, listing interactive elements with their names, roles, and positions. If the tree lacks labels, fall back to OCRing the window and matching on visible text. Screenshot analysis should be reserved for canvases, icons without names, or situations where the tree cannot provide sufficient information. After each critical action, re-read the tree or text to verify that the click was successful. Ultimately, this approach minimizes costs and maximizes precision, while still accommodating apps that lack proper accessibility information.",
  "summary": "Every computer-use agent has to answer one question before it can click anything: what is on the screen right now? There are two ways to answer it. Send the model a screenshot and let it find the button in the pixels. Or ask the operating system, which already knows: every serious UI toolkit publishes an accessibility tree for screen readers, with the role, name, state and position of each…",
  "key_points": [
    "Accessibility tree provides element roles, names, states, and positions.",
    "Screenshot method clicks guessed coordinates, infers details.",
    "Accessibility tree can be incomplete, screenshots useful backup."
  ],
  "editors_take": "Combining accessibility trees with visual screenshots allows agents to reduce token costs and increase clicking precision while maintaining the ability to navigate complex applications that lack structured metadata.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}