Cosmos 3 Edge, Part 2: Physical AI Workflows, Action Reasoning, and Action Chunks
In Part 1, we introduced NVIDIA Cosmos 3 Edge and its deployment on Jetson. Traditional vision AI asks, “What objects are in front of me?” Physical AI goes further: What is happening now? What could happen next? If I take an action, how will the world change? World models aim to connect objects, space, motion, actions, and outcomes. World Action Models (WAMs) extend that relationship toward robot…
In Part 2, we delve deeper into NVIDIA Cosmos 3 Edge's capabilities, specifically focusing on physical AI workflows, action reasoning, and action chunks. Traditional vision AI is limited to identifying objects, while physical AI takes it a step further by understanding the actions and consequences of those actions. Cosmos 3 Edge, a 4B parameter model from NVIDIA's Cosmos 3 family, extends this capability to robot behavior through world models and world action models (WAMs).
World models aim to connect objects, space, motion, actions, and outcomes, while WAMs specifically focus on robot behavior. The article discusses how Cosmos 3 Edge moves from "Can the model run?" to "Can it help decide what to do next?", emphasizing the importance of action chunks in connecting vision, spatial reasoning, motion, and outcomes for robot behavior.
An action, in the context of robotics, is a numerical representation of movement over time. For example, "move your hand to the right" is not directly executable motor control. Instead, robot commands are numerical representations over time, which can encode translation, rotation, joint targets, or gripper state. In Cosmos 3 Edge, the output can include an action trajectory, which contains a sequence of targets at successive times.
The article highlights the importance of a feedback loop in physical AI, where observation leads to understanding the scene and task, predicting possible outcomes, choosing and executing actions, and then observing the changed world again. This closed loop is crucial for robots to learn and adapt to their environment.
To enable efficient control of robots, the article introduces action chunks. Rather than request a fresh large-model inference for every tiny movement, the system predicts a segment of motion and lets the controller execute it. This approach, known as chunking, reduces the number of large-model calls required for low-level control updates. The Jetson AI Lab guide demonstrates this pattern using 640 × 360 observations on Jetson Thor, with repeated replanning to stay aligned with the task.
The deployment of Cosmos 3 Edge at the edge, such as on Jetson AGX Thor, offers several advantages, including reduced network latency, bandwidth requirements, and dependence on the connection. The system can generate action chunks approximately 1.53 seconds before the current motion finishes, allowing for a window to prepare the next chunk and maintain continued motion and replanning.
However, it is essential to note that deploying a model and plugging in a robot does not automatically result in an autonomous worker. The article cites a 22.9% success rate for a post-trained policy on 120 language-conditioned manipulation tasks, which belongs to a particular benchmark and configuration. In conclusion, Cosmos 3 Edge's physical AI capabilities enable robots to perceive, understand, predict, and act in their environment, paving the way for more advanced robot behaviors and on-device control.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.