What we learned giving AI agents their own mouse and keyboard

At Novastart, an infinite canvas operating system was developed, allowing every desktop application to exist on a single endless canvas. This canvas is capable of restoring workspaces exactly as they were left, providing a seamless user experience. The company then introduced AI agents to work on this canvas, each assigned their own mouse and keyboard. The three key takeaways from this experiment are as follows:

1. A single lead agent is responsible for drafting a plan, assigning one task for each of the other agents, and opening a window for each task. This organization proves manageable, as up to 10 agents per task can be accommodated without issue, as each agent maintains a visible presence on the canvas.

2. While AI agents can operate efficiently, human intervention remains crucial. The system allows humans to step in at any time, freezing everything with a simple pause. This pause feature enables users to type corrections mid-run, and agents will adjust their actions accordingly, eliminating the need to restart or manually return windows to their previous positions.

3. The definition of "done" is essential to ensure that agents accurately report successful outcomes. If an agent fails to achieve its goal, it will report the lack of success. To improve reliability, a separate check was implemented, comparing a fresh screenshot before marking a task complete. This additional verification process emerged as the most significant reliability improvement.

In addition to the lead agent and its helpers, the canvas supports multiplayer functionality. This means that teammates can share the workspace, each with their own cursor and access to chat and other apps. By leveraging this shared workspace, collaboration and communication among team members are streamlined, enhancing overall productivity.

As the company continues to explore the use of AI agents in computer use, they are actively seeking patterns to ensure the agents' honesty and reliability. By learning from the successes and challenges encountered during this experiment, Novastart aims to refine their approach and further optimize the integration of AI agents in their operating system.