How AI ‘world models’ are helping robots learn to adapt in the real world

Artificial intelligence systems are being created to help robots adapt to new situations by utilizing "world models." These systems aim to provide machines with an understanding of their surroundings, enabling them to anticipate changes and determine appropriate responses. Most existing robots are trained to carry out pre-defined tasks in familiar settings, but the real world is constantly evolving, making it challenging to program a machine for every conceivable scenario.

ACE Robotics, a Chinese firm, is among several companies developing this technology. Chairman Wang Xiaogang explained that traditional models are limited to specific, simple tasks. However, with a world model, a robot can comprehend how the world has changed and apply that knowledge to execute new tasks.

World models do not merely learn what action to take in a given situation; instead, they aim to build a comprehensive understanding of the environment. This entails predicting potential changes and utilizing that information to inform subsequent decisions. As AI progresses from the digital realm into physical environments, the importance of such capabilities becomes increasingly evident.

Wang noted that while large language models specialize in answering questions and coding agents within the digital world, the next step is to extend these capabilities to physical environments, thereby improving productivity.

Teaching machines to comprehend the physical world necessitates vast amounts of data. Large language models can be trained using extensive textual information, but world models require data on physical movement and interactions between people and objects. ACE Robotics estimates that approximately 10 million hours of data might eventually be necessary – roughly one hundred times the current data available.

To acquire additional data, ACE Robotics is collaborating with various businesses, shops, and factories to record humans carrying out daily activities. Participants wear sensors, and cameras capture their movements, such as hand and finger positions. This information can subsequently be utilized to instruct robots on performing similar tasks.

In retail environments, for instance, ACE Robotics can record a person picking up items and placing them in a basket, subsequently extracting information about the individual's body movements and hand actions. Mistakes also provide valuable training data. If someone drops an object while carrying out a task, the system can learn not only the proper method of completing the task but also the appearance of failure, according to Wang.

However, for certain applications, particularly in intelligent-driving systems, the situations that machines must learn from may be the most difficult to encounter in real life. Huawei, a Chinese tech giant, employs a comparable approach to its Qiankun ADS 5 intelligent-driving system. This system combines real-world driving data with a virtual "world engine," allowing different traffic scenarios to be generated and used to train the system.

Over 25 car brands utilize Huawei's intelligent-driving technology, granting it access to an expanding pool of driving data. Additionally, simulations enable developers to generate more challenging or unusual scenarios that may occur infrequently in reality, providing sufficient training data. Regardless of the training data source—whether from humans, vehicles, or simulations—the ultimate objective is to develop AI systems capable of adapting to an unpredictable physical world.