Into the Omniverse: How Developers Turn Ideas Into Simulations With Frontier AI Agents

Developers are leveraging frontier AI models in collaboration with NVIDIA Omniverse libraries to transform simulation ideas into functional applications. By providing natural-language instructions, developers direct AI agents to connect physics, rendering, and sensor simulation capabilities, review results, and guide necessary changes. NVIDIA's Omniverse libraries grant access to GPU-accelerated physics, rendering, and sensor simulation features.

One example is a humanoid simulator for a warehouse environment, where Frank DeLise, an Omniverse product manager at NVIDIA, utilized Astra, an AI agent, to build an interactive simulation. He directed Astra to integrate NVIDIA's physics (ovphysx), scene updates (ovstage), rendering (ovrtx), and UI (ovui) libraries. DeLise also used Astra with SimReady (simready-foundation) to generate both physical scenes and related animation, rendering code.

Another application involves using frontier AI models for autonomous driving simulations. Doyub Kim, a simulation technology team manager at NVIDIA, instructed Astra to create a reusable simulation environment based on San Francisco's Market Street. Kim guided Astra to map out the workflow, connect asset creation, traffic, Omniverse RTX sensor simulation, and Alpamayo driving in stages, and check each integration.

The prototype served as a testing ground for comparing driving models and understanding how scene or sensor modifications impact autonomous vehicle behavior.

For digital twin creation, Ashley Reid, an RTX sensor validation engineer at NVIDIA, used Astra and Claude Fable 5 agents to compare recorded and simulated camera and lidar outputs. The agents generated two digital twins from scratch and improved two existing ones, refining them based on camera and LiDAR metrics. Reid directed Astra and Claude Fable 5 to measure discrepancies between recorded data and simulated sensor outputs, creating or modifying OpenUSD scenes as necessary.

The iterative workflow concluded with a more accurate digital twin representing the real environment.

Lastly, NVIDIA's Robo Olympics project demonstrates how AI agents can assist in testing robotic skills. Tae Kim, who leads NVIDIA Omniverse engineering and product, employed natural-language instructions and sports videos to guide Astra in building simulations for a humanoid robot to perform sports movements. Utilizing the Newton Physics Engine, NVIDIA Warp framework, and ovrtx renderer, Kim directed Astra to build controllers and refine them through physics trials.

In one experiment, the robot cleared a hurdle in 64 out of 100 simulation trials, providing Kim with valuable feedback for enhancing the robot's timing and control.