Meta AIRA: From Training Ground to Kaggle Gold: A Step-by-Step Look at What Happened

Meta's AIRA project has transitioned from simulated training to winning a gold medal at Kaggle's competition. This transformation involved moving from a single arena to one with multiple environments. The research, conducted by Facebook AI Research (FAIR) team at Meta, began with agent migration to the training arena and culminated in a top-tier prize from a Kaggle competition with around four thousand participating teams.

The evaluation metrics for this competition are taken from Kaggle's real-world challenges, including a ranking system based on the number of gold medals earned. The experiments largely ran on MLE-bench lite, a subset of 22 challenges known for their lower difficulty. The key metrics tracked include the medal rate, or the proportion of challenges an agent can solve gold medal level or higher.

The first stage of improvement came from changing the arena, resulting in a 10.7-point increase in performance on the lite set using the model o1-preview. Moving to the second stage, the researchers adjusted the agents' thinking and decision-making processes. This included modifying operators, which are the basic commands agents use to perform tasks, and search policies, the strategies used to decide the next step.

Operators were reconfigured to adjust complexity based on the situation, generate smaller outputs when options were limited, and adapt to the situation for debugging. A different search policy was tested, including Greedy (choosing the best option available), MCTS (Monte Carlo Tree Search), and Evolutionary (adapting solutions through genetic algorithms). The best combination was found to be AIRA using Greedy operators on the DeepSeek-R1 backbone, achieving a 47.7% medal rate on the lite set.

The final stage involved scaling the environment to multiple GPUs and expanding the challenge set to 50 diverse problems, including 20 new research-style challenges. The key to success in this stage was the use of percentile rank as the primary evaluation metric, which measures an agent's performance relative to human leaders. AIRA3 achieved an impressive percentile rank of 81.5 on MLE-bench-30, showcasing significant improvements across various components of the system.