Your AI agent can plan anything. It still can't click "Post".
Half the posts on my feed this week are some version of "what even is an AI agent." If-statements in a trench coat. Memory vs. RAG. Verification bottlenecks. All good debates — and all about the agent's brain . Nobody's talking about its hands . Here's the wall I kept hitting building an autonomous agent: the model can decide, perfectly, that it should reply to a lead, check the company mailbox,…
This week, many social media posts have been discussing "what even is an AI agent." Some conversations revolve around if-statements, memory versus Retrieval-Augmented Generation (RAG), and verification bottlenecks. However, the conversation that's lacking is about the agent's "hands." My article aims to address this gap. The core problem I faced while building an autonomous agent is that the AI model can decide perfectly well on which action to take, but it cannot actually execute it.
The LLM generates text, but it doesn't "click Post" on a logged-in account. To overcome this limitation, I developed a solution called Ghost Browser. This tool operates like a human, using a real Chromium browser to interact with websites. It sees the webpage rather than parsing it, ensuring that the agent can handle any redesigns without breaking.
By painting over interactive elements with numbered boxes and taking a screenshot of the page, the agent receives an annotated picture plus a numbered list, enabling it to perform actions even when selectors change or are unavailable. Furthermore, I built this system with a login-one-and-drive-forever approach, where the agent operates a real, logged-in session.
This eliminates the need for re-authentication and protects against account lockouts. To ensure safety and control, an "act-gate" is implemented, requiring human approval before the agent can perform actions like posting or messaging. This gate exists twice, enforced by the prompt and a guard inspecting the label of any action the agent is about to take.
The design of Ghost Browser also includes additional features such as a Tailscale exit for a residential IP, an SSRF guard to prevent server-side request forgery, and a workflow engine with route cards for deterministic flows and replaying site traffic without relying on the model. With the codebase open-sourced under the MIT license, I invite designers to contribute and improve the console UI, as the engine is robust and production-tested.
You can find the repository, setup guide, and runnable examples at https://github.com/Wvdstoep/ghost-browser. I'm curious to hear your thoughts on whether "hands" should be considered the missing piece in the agent stack or if there are alternative solutions to this problem.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
