We’re announcing Genie 4, our new world model for real-time interactive simulation.

Google DeepMind unveiled its latest world model, Genie 4, in a research preview on Tuesday. This advanced model produces highly photorealistic and interactive environments, allowing users to modify the scene mid-simulation by typing prompts. A distinctive feature of Genie 4 is its ability to generate real-time audio that aligns with the simulated world, encompassing music, ambient sounds, and interactive dialogue.

This latest release occurs a few months after the launch of Runway's second world model, GWM Worlds 2, which also supports dynamic user inputs and audio generation.

The world model market is starting to unify, with major players like AMD, Amazon, and Rivian making moves to bolster their simulation capabilities. Google's Genie 4 is a strategic response to this competitive landscape. Released as a research preview over a year after Genie 3, this new model is even more photorealistic and incorporates audio.

It also supports freeform edits such as altering weather conditions, camera angles, adding new elements, and maintains a consistent scene for up to two minutes – twice as long as its predecessor. Moreover, Genie 4 boasts improved physics and more human-like characters.

Google's AI model, Gemini, is tightly integrated with Genie 4, functioning as the planner that maps out steps and validates frames while Genie generates the visuals. This synergy was demonstrated in a research experiment involving Gemini Robotics, where a Gemini 4 Argon model broke down tasks into sequential steps that Genie 4 simulated in real time. This experiment hints at the potential for testing robot scenarios in simulated environments before actual deployment.

Tejas Kulkarni, Google's research director, discussed how we may increasingly encounter hybrid world models in the future, combining Gemini's text and code generation abilities with Genie's real-time simulation. Potential commercial applications span gaming, interactive learning, and robotics. One intriguing possibility is using Genie 4 to delve into interactive learning, such as engaging with characters in a historical painting from a different era.

However, some trusted testers speculate that Genie 4 could also revolutionize film production. Diego Rivas, a Google DeepMind engineer, highlighted the model's real-time capabilities as especially beneficial for rapid ideation and storyboarding phases in the creative process. He also mentioned that Gemini Omni, a multimodal large language model, could eventually convert these prototypes into creative assets, though this is still a long way off.

Rivas emphasized that Genie 4 is still in its early stages, and the ultimate end-to-end product experience remains uncertain. The team acknowledged existing limitations, including drift in details and objects over time, and challenges in ensuring accurate audio and legible in-world text. Genie 4 is now available as a research preview for trusted testers, with plans to extend access to Google AI Ultra subscribers, Project Genie users, Gemini API early adopters, and Google AI Studio participants.