Caterpillar and CoreWeave shorten the learning loop for physical AI

Physical AI is gaining traction as a solution to enable machines to perceive and react to their environments. Caterpillar Inc., an expert in autonomous mining equipment, aims to bring this technology to construction. However, this transition presents significant challenges. Brandon Hootman, Caterpillar's vice president of physical AI platforms and construction autonomy, explained that while mine sites change, construction sites are far more dynamic and variable.

Richard Ahlfeld, senior vice president of Physical AI at CoreWeave Inc., and Hootman discussed the hurdles of integrating physical AI into construction during an interview on theCUBE Research's Dave Vellante and John Furrier show at the Fully Connected event. CoreWeave recently introduced a Physical AI Field Engineering service, pairing its engineers with customers' domain experts to streamline the learning process for construction machines.

This collaboration involves ingesting telemetry and vision data, simulating digging scenarios, and applying reinforcement learning. Caterpillar already stores approximately 18 petabytes of federated data from its machines, dealers, and customers. Nevertheless, training autonomous equipment demands even more data, including perception data synchronized with machine control and performance metrics.

Hootman noted that this data amounts to terabytes of information daily for a single machine. CoreWeave leverages Caterpillar's data, along with GPU capacity and expertise from Nvidia, to annotate and label incoming field data. This enables a reduction in the feedback loop time between data collection and utilization, from months to hours, thereby accelerating the iterative development process.