Has a Chinese physical AI start-up manipulated a global ranking to beat Nvidia?

A Chinese physical AI startup, Spirit AI, briefly claimed global leadership in robotics by briefly topping RoboArena, a global benchmark for physical AI. The benchmark, co-developed by Nvidia and institutions such as Stanford University and the University of California, Berkeley, evaluates generalist robot policies – the software guiding movement and execution.

However, Spirit AI's victory was brief as the benchmark's creators later removed Spirit's model and several others from the official list, citing "benchmark hacking". Spirit AI generated significant buzz, dubbing RoboArena the "Olympics" of embodied intelligence in North America. The controversy highlights vulnerabilities in RoboArena's original design, which allowed anyone to evaluate models and conduct comparative studies.

This led to potential gaming of the system, with about half of Spirit AI's evaluations conducted by a single account named ECUST Robot Lab. The evaluations were later deleted, and it remains unclear if the account was affiliated with Spirit AI. To curb manipulation, RoboArena updated its methodology, restricting testing to vetted third-party volunteers and introducing an "official" leaderboard requiring a minimum of 100 verified evaluations.

Spirit AI's model was removed from the official list due to unfulfilled sample size requirements. Nvidia's model, DreamZero, now tops the official list, while Spirit AI's model dropped off the unofficial ranking due to a low sample size.