The Agent Left the IDE

The most intriguing aspect of AI coding agents is their ability to operate computers, not merely generate code. This shift in functionality transforms the way software development workflows operate. Instead of simply writing code, these agents can now inspect the app, click through broken flows, read console logs, run servers, and verify whether the system works.

A recent update from OpenAI's Codex on May 29th has further enhanced this capability by enabling computer use on Windows within the Codex app for eligible users. This update allows the agent to see, click, and type within Windows applications while testing and refining software. This new shape of work emphasizes that coding is not just about typing, but about operating the complete system around the code.

The IDE, while a natural starting point for AI coding tools, is insufficient as it only provides a partial view of the product surface. Instead, the agent needs to be able to observe, diagnose, change, and verify across various platforms like the browser, terminal, database, logs, design tools, cloud dashboards, and mobile simulators.

This new form of remote control in the coding process can change the cadence of engineering. Instead of a synchronous, stop-start coding process, we can see a supervisory model where the agent runs independently while the human oversees and intervenes when necessary. However, this increased autonomy also brings potential risks. The agent could inadvertently click the wrong button, misunderstand a modal, or test against the wrong environment, leading to unintended consequences.

Therefore, setting clear permissions, using disposable environments, and implementing human approval for risky actions are essential. Ultimately, the best workflows will involve agents with transparent work processes, allowing for easy inspection of their actions, updates, and verification trails.