Inside OpenAI's plan to make ChatGPT the way you use your computer

OpenAI, the artificial intelligence research laboratory, is set to revolutionize the way we interact with our computers by incorporating Computer Use tools into its ChatGPT platform. This move is part of the company's broader strategy to automate more work for its users.

The ability of ChatGPT to perform actions in computer applications and web browsers is a significant leap forward for the AI industry. OpenAI's staff have been working tirelessly to perfect this technology, which has been a long-standing ambition since the company's founding in 2015. The challenge lies in collecting and creating the necessary data to train these AI agents, but OpenAI now believes it has enough confidence to roll out these tools to its customers.

In a bid to make these tools more accessible, OpenAI launched a Chrome extension in 2023 that allows ChatGPT to take control of the browser. Additionally, cloud and in-app browsers have been created to enable ChatGPT to interact with public websites. The company's Codex coding tool has also been equipped with Computer Use capabilities, making it possible for users to have ChatGPT complete tasks on other apps.

OpenAI's president, Greg Brockman, revealed that the April update made their technology available not just for coders but for anyone who performs computer work. Employees working on these tools at OpenAI shared that while there is room for improvement, the tools are at an inflection point.

Anthropic, a rival in the AI industry, is also racing to improve its version of Computer Use, having been the first to market in 2024. As of May, at least 600,000 organizations had tried Anthropic's Claude Cowork feature, which utilizes the same Computer Use technology.

OpenAI's Computer Use team manager, Ari Weinstein, believes that once ChatGPT can use computers and software faster than humans, it will significantly alter how users interact with their devices. He highlighted that users are likely to offload more tasks to ChatGPT as its capabilities improve.

The training of OpenAI's newest models for Computer Use involved several phases. Initially, the models were built with vast amounts of data, including frame-by-frame screenshots. Subsequently, developers provided the models with data showing the ideal inputs that lead to specific outputs, such as filling out a tax form or 3D-modeling an object. This data often comes from human trainers, who demonstrate how to complete tasks, a process known as reinforcement learning.

OpenAI has also improved how its models interact with computers and process data in real-time. Previously, ChatGPT took screenshots of users' computers, analyzed pixels, injected commands, and repeated the process. Now, when a website is open, the tool can swiftly read the page's memory structure, accessibility information, and link connections. The tool still takes screenshots, but users are asked to allow ChatGPT to record their screen during setup, enabling it to rapidly analyze what's on-screen.

This combination of tools provides OpenAI's Codex with a way to test the code it builds, find bugs, and make improvements in a continuous loop. For hobbyist coder Cristian Medina Ruiz, this technology has been a game-changer, enabling him to rapidly iterate and verify his code even when he's away from his computer. OpenAI aims to leverage ChatGPT to interact with the open internet, with the potential to automate data entry work, compliance tasks, and calendar scheduling.

As the technology continues to evolve, it remains a work in progress, with Yu, an AI researcher at Columbia University, noting that there is still a significant gap between the technology's current speed and what consumers want in terms of generalization to any webpage, application, or operating system.