Giving an AI Agent a Real Phone: What ADB Automation Actually Looks Like

The browser was easy for me to control with my agent, but the phone proved humbling. My agent already handled a desktop environment, so it seemed straightforward to point it at an Android device using ADB. However, it took much longer than expected - two weeks of work, with every challenge I faced mirroring the lessons learned from the desktop experience. The interface proved easy to navigate, but the state management proved incredibly tricky.

Using ADB, I was able to retrieve the view hierarchy as XML with bounds for each node, inject events via adb shell input tap/swipe/text, and capture screenshots with adb shell screencap -p. While this foundation allowed me to drive any app a human could operate manually, it also introduced new challenges. Three major obstacles stood in my way:

1. Screens with FLAG_SECURE protection, such as password fields, banking apps, and some onboarding flows, returned zero-byte screenshots. If my loop treated an empty image as an unchanged screen, it would continuously tap randomly for extended periods. To overcome this, I checked the byte length and PNG header, treating a secure screen as a signal to switch strategies.

2. The UI tree often misrepresented which elements were tappable. Jetpack Compose and Flutter apps frequently presented a single node covering the entire screen, causing every coordinate to resolve to the same element. The solution was not to use larger coordinates, but to utilize the keyboard (adb shell input keyevent) to move focus between fields. After each action, I verified the tap by reading the tree again instead of assuming it had landed correctly.

3. The gettext command did not exist in the shell. To input non-trivial text, I resorted to using am broadcast to set the clipboard content, followed by input keyevent 279 to paste. Typing Chinese or any non-ASCII characters via input text proved problematic. The key rule that saved the project was never to act on a cached view of the screen.

I needed to read, act, and read again. Although this approach was slower per step, it significantly accelerated the overall task completion time, as I no longer discovered errors five steps later that the third tap had gone to the wrong window.

With these techniques in place, driving the phone became manageable. The agent, which previously required manual effort for repetitive tasks, could now open apps, log in once, and perform the necessary actions. This achievement demonstrated that what initially appeared as a simple weekend project evolved into a complex endeavor requiring careful consideration of the state management challenges inherent in automating a real phone using ADB.