Android Bench 2.0 focuses on long-horizon tasks, agent evaluations

Google has unveiled Android Bench 2.0, a development tool aimed at assessing complex AI tasks beyond simple bug fixes or minor updates. This new version zeroes in on long-horizon tasks that typically consume multiple days or even a week for human engineers to accomplish, such as creating new features, constructing apps from scratch, and converting cross-platform software to Android.

To better evaluate the performance of AI agents handling these intricate assignments, Android Bench 2.0 employs a more sophisticated scoring system. It now provides continuous scores instead of simple pass or fail ratings, taking into account factors like functionality, visual appearance, and the avoidance of regressions. The system also imposes penalties for any deviation from the evaluation guidelines or structural constraints.

To gauge the prowess of various AI models, Google has benchmarked them using Android Bench 2.0. The results show that GPT-6 Astra leads the pack with a 28% pass rate, while other models like Gemini 3.7/3.8 Flash, OpenAI's GPT-6, Anthropic's Fable 5.1, Kimi K3, and Qwen 3.8 Max trail behind, with scores generally in the high 90s from the previous benchmark's perspective.

The report highlights that converting cross-platform apps to Android remains a demanding challenge, as no AI model has achieved a flawless 100% pass rate. In fact, even the most advanced frontier models typically manage to complete around 80% of such tasks. During testing, Android Bench 2.0 ran agents provided by the respective model creators, such as Gemini 3.8 Flash on Google's Antigravity platform and GPT-5.6 Sol with Codex.

Google's team believes that the design of these agent evaluations significantly influences the overall developer experience. As such, they plan to incorporate a wider range of models and agent combinations in future iterations of Android Bench 2.0. Stay tuned for more updates on this development at 9to5Google on YouTube.