Embedded AI evaluators won’t be an easy solution, Hugging Face’s policy chief says

Hugging Face's chief policy officer, Irene Solaiman, cautioned that embedding AI evaluators will not be a simple fix, despite the growing interest in this approach. Model benchmarking has become a cornerstone of AI labs' reputations, but the evaluation methods are complex and still developing. To build trust in these tests, reproducibility is key, as it requires access to the model itself, which is best achieved through open weights.

However, comparing different evaluations remains a challenge, as they are an evolving science and lack a well-established ecosystem for meaningful comparisons. The concept of embedded evaluators — neutral third-party observers placed within AI organizations — has gained traction as a potential solution to address safety concerns. Yet, doubts persist about the neutrality and effectiveness of these evaluators.

Recently, Anthropic's decision to hire Accenture as its first evaluator sparked controversy, with many criticizing the consulting firm's qualifications for the task. Evaluations are now shifting from assessing models alone to evaluating the entire system, including the harness and agent. The harness, or the tool users interact with to access the model, plays a significant role in determining performance. A less powerful model with a better harness could outperform a more sophisticated model with a weaker harness.

Despite the growing number of evaluations, there are relatively few for harnesses, and the development of comprehensive evaluations for the holistic agent is lagging. In summary, the evaluation landscape is currently fragmented, and for benchmarking to remain a valuable tool for industry leaders, a more standardized, industry-wide approach may be necessary.