I Saw the Future of AI in a Robot That Can Learn on the Spot

In the Cambridge, Massachusetts offices of Generalist AI, a startup specializing in artificial intelligence, I observed robot arms executing mundane tasks with remarkable speed and adaptability. Astonished by their performance, which eerily resembled human intelligence, I witnessed the robots swiftly mastering various activities after merely watching a brief instructional video. Notably, they were able to adapt to changes in their environment without additional training.

One striking demonstration involved a robot tasked with sweeping a block into a bowl using a dustpan and brush. When the brush was removed, the robot ingeniously substituted it with the dustpan, successfully flicking the block into the bowl. In another instance, a two-armed robot watched a video of someone unzipping a purse and removing money. It replicated this action with a different purse, demonstrating an impressive ability to adapt to new situations.

The robot's improvisational skills were particularly remarkable, such as switching to its left gripper when it couldn't effectively grab money with its right one. Generalist AI's co-founder and CEO, Pete Florence, compared this progress to the groundbreaking capabilities of OpenAI's GPT-3, highlighting the potential for robots to rapidly learn new tasks and transfer their knowledge across various scenarios.

This innovation may stem from the company's focus on teaching robots about the physics of the world, inspired by the intuitive understanding of physics that humans possess from a young age.

Generalist AI's researchers were often surprised by the robots' improvised solutions, such as using a banana to sweep items in front of them. This highlights the critical need for physical intelligence in machines and offers valuable insights for AI researchers, drawing parallels to how children efficiently learn about their surroundings.

The company's founders, Pete Florence, Andrew Barry, and Andy Zeng, all previously worked at leading AI and robotics companies, bringing a wealth of expertise to their venture. Traditional AI training methods, which involve feeding thousands of examples into the model, are notoriously imperfect and often fail when changes, like lighting, are introduced.

Generalist AI aims to address these limitations by developing a general robotic model trained by humans, using special grippers with cameras attached to them that workers can manipulate to perform tasks.

While the company remains tight-lipped about their exact training methods, they have amassed a vast amount of high-quality data. Unlike other companies focused on more specialized robots, Generalist AI has created their AI models from scratch, rather than relying on open-source language models. According to Danfei Xu, a roboticist at Georgia Tech, Generalist AI stands out for its ambitious approach and exceptional robotic skills.

Xu believes their work is nearing deployment in commercial settings, particularly due to their data-driven approach, which collects interaction data at a large scale without being overly tied to a specific robot model.

However, Generalist AI acknowledges that their learning algorithms are still not entirely reliable. On average, the robots can only complete tasks shown to them about 59 percent of the time, indicating room for improvement before achieving near-perfect success rates. Additionally, it remains unclear how well these skills will generalize to an extensive range of tasks and environments.

Despite these challenges, the potential for robots to quickly learn and adapt to new skills in industries like manufacturing is immense. A recent demonstration showed an engineer using a two-armed robot as a makeshift assistant, with the robot surprisingly joining in to help stack cups. The engineer's delight at this unexpected collaboration exemplifies the exciting possibilities that lie ahead for AI-powered robots.