Your computer-use agent clicks 20 pixels off: screenshots vs the accessibility tree

When a computer-use agent needs to click an element on the screen, it must first determine what is currently displayed. There are two ways to accomplish this: send the model a screenshot to analyze the pixel grid, or query the operating system's accessibility tree, which already contains the role, name, state, and position of each control. Most demos use screenshots because they work universally, but a combination of both methods yields better results.

The accessibility tree provides several advantages over a screenshot. It clicks the actual element rather than a guessed coordinate, informs the agent of the button's state (e.g., disabled, checked, focused), and eliminates the need for the model to infer details like a greyed-out button from a slightly lighter shade. Moreover, a tree is cheap to generate; a screen with 40 controls typically requires only a few dozen short lines of text, while a screenshot costs roughly width x height / 750 tokens per step.

However, the accessibility tree can be incomplete or misleading. Canvas apps, design tools, remote screens, Electron and Chromium applications, and custom-drawn controls with no names can all produce an empty or inaccurate tree. Similarly, stale positions during animations and scrolling can cause issues. In these cases, screenshots remain a useful backup option.

To ensure reliable performance, it is essential to understand how each approach works and where they might fail. The accessibility tree can be inspected using tools like the Accessibility Inspector on macOS, Accessibility Insights for Windows, or Accerciser on Linux. For Android devices, adb shell uiautomator dump can be used to generate the view hierarchy.

The recommended workflow is to prioritize the accessibility tree, listing interactive elements with their names, roles, and positions. If the tree lacks labels, fall back to OCRing the window and matching on visible text. Screenshot analysis should be reserved for canvases, icons without names, or situations where the tree cannot provide sufficient information.

After each critical action, re-read the tree or text to verify that the click was successful. Ultimately, this approach minimizes costs and maximizes precision, while still accommodating apps that lack proper accessibility information.