Verifiable AI: Proving What Happened Behind the Black Box

The article discusses the challenges of verifying AI model outputs without compromising proprietary information. Researchers at UC Berkeley found that classifiers trained to distinguish between quantized and full-precision model outputs scored only around 50 percent, indicating the difficulty in verifying model behavior. A statistical test on larger models only caught substitution reliably when about 60 percent of queries were swapped, and outputs for the same open model matched across different providers less than 5 percent of the time when sampling was switched off.

This lack of verifiability poses risks for decision-making processes that rely on AI models, as customers have no way to check if they are receiving the correct model outputs.

Three potential solutions to address this issue are presented: zero-knowledge proofs (zkML), optimistic machine learning (opML), and trusted execution environments (TEE). zkML relies on mathematical proofs, where the model owner publishes a commitment to the model's weights and generates a proof alongside the output. This allows anyone to verify that the committed model ran without revealing the weights.

However, this approach is currently too slow for large models and incurs a significant computational cost. opML, on the other hand, relies on economic incentives, where the provider posts the result with a deposit and the result is treated as correct unless challenged. This approach is faster but relies on the honesty of the provider and the availability of the weights for a challenge.

TEEs rely on hardware-based attestation, where the chip manufacturer signs a report listing the loaded code and model. This provides some level of trust in the hardware but is still susceptible to side-channel attacks and relies on the trustworthiness of the hardware vendor. The article concludes by highlighting the potential applications of these verification methods in regulated sectors, such as lending, medical diagnosis, and insurance pricing, where small models are commonly used for high-stakes decisions.