Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation
Evaluating an AI system requires disaggregated assessment, as performance varies across domains such as benchmark task types or conversation types in deployed agents. Exhaustive testing is expensive, so evaluation rests on a sample of labeled units. We treat the evaluation set as a finite population and seek accurate point and interval estimates of each domain mean. Direct estimators, including…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
