New Methods for Disaggregated AI Evaluation

What changed

This research introduces novel methods for disaggregated AI evaluation, addressing the challenge of performance variability across different domains such as benchmark task types or conversation types in deployed agents. The authors propose Prediction-Powered Smoothing (PP-S), a Bayesian model that smooths prediction-powered estimates within each domain. An extension, Prediction-Powered Taxonomy Smoothing (PP-TS), further enhances this by borrowing strength across a reporting taxonomy. For validation, a new, approximately unbiased design-based cross-validation score is derived to aid in selecting the best estimator among direct and smoothed options. The proposed estimators were evaluated on a curated benchmark with verifiable grading and on deployed agent traffic graded by humans. In both scenarios, the new estimators demonstrated improvements over direct estimators in terms of point and interval estimation accuracy, achieving near-nominal coverage rates. Furthermore, at equivalent sampling budgets, the proposed cross-validation score performed comparably to an independent validation sample in selecting estimators and provided significantly more accurate estimates of the selected estimator's error.

Why it matters for builders

As AI systems become more complex and are deployed across a wider array of applications, understanding their performance in specific contexts is paramount. Traditional evaluation methods often rely on a limited sample of data, which can lead to imprecise performance estimates, particularly for specialized domains. The PP-S and PP-TS methods offer a way to derive more robust and reliable performance metrics by leveraging data from related domains. This allows builders to gain a clearer picture of their models' strengths and weaknesses across different use cases, facilitating more targeted improvements and better deployment strategies.

Practical impact

Builders can explore the application of PP-S and PP-TS to their own evaluation pipelines. When evaluating models on benchmarks or deployed systems, consider implementing these smoothing techniques to enhance the precision of performance estimates, especially for domains with sparse data. The derived cross-validation score can be used to systematically choose between direct and smoothed estimators, optimizing the evaluation process. This approach can lead to more accurate uncertainty quantification, providing better confidence intervals for performance metrics. The research suggests that these methods can achieve better estimation accuracy with the same sampling budget compared to traditional approaches.

Caveats and source limits

The research is presented as a pre-print on arXiv, and details regarding implementation availability or specific benchmark results beyond the described studies are not provided. The paper focuses on the theoretical framework and validation of the proposed estimators using curated datasets. Further independent benchmarking and real-world application studies would be beneficial to fully assess the practical scalability and effectiveness of PP-S and PP-TS across a broader range of AI systems and evaluation scenarios. The exact computational costs and resource requirements for implementing these methods are not detailed.