Piloting the world's first double-blind AI evaluations

Ensuring reliable evaluations of advanced AI models requires preventing them from "peeking" at the test questions before the assessment, a problem known as benchmark contamination. To address this challenge, Google has introduced the industry's first double-blind evaluation of a proprietary, frontier-class AI model called Gemini Flash Lite.

This innovative approach safeguards both the evaluator's test prompts and the proprietary model weights through cryptographic means. By using Google Cloud's Confidential Computing portfolio, the evaluation process creates a secure "box" where external testers and the model provider cannot access each other's sensitive data. The Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons are collaborating with Google to test the Gemini Flash Lite model against confidential benchmarks in a privacy-preserving environment.

This pilot project aims to establish a new standard for model oversight, promoting trust and reliability in AI systems for high-stakes applications such as cybersecurity and government use cases.