The Next Evolution of AI Is Learning From Your Dodgy Gaming Skills

The future of artificial intelligence may lie in learning from human gaming skills, as a new startup called Worldmodeldata aims to address the limitations of large language models. These models, trained solely on text, struggle with tasks requiring physical dexterity and precision, such as piloting autonomous vehicles or operating robotic arms. To overcome this, researchers like Fei-Fei Li and Yann LeCun are developing "world models" trained on visual and action data, specifically designed to understand real-world physics.

Unlike traditional language models, world models require cause and consequence data, which is scarce on the internet. Worldmodeldata hopes to fill this gap by collecting controller inputs and other data from video game studios, who generate vast amounts of such information. The startup, advised by Yann LeCun, curates and organizes this data, saving researchers the trouble of negotiating individual agreements with numerous game studios.

The theory is that as the size of the training datasets for world models grows, their performance will improve similarly to that of large language models. However, the availability of suitable training data remains a significant bottleneck. While some companies collect game data internally, Worldmodeldata offers a shortcut by aggregating data from millions of popular video games, presenting a more diverse and abundant source than other potential sources.

Despite the promise of video game data, some experts remain cautious. Nvidia, for example, uses a custom engine to create world models optimized for real-world physics, arguing that video game physics often lack fine-grained control. Nevertheless, researchers remain optimistic, believing that video game data could eventually make up the majority of training material for world models, leading to breakthroughs similar to the development of GPT for language models.