New Method Optimizes Data Annotation for Off-Policy Evaluation

What changed

This paper introduces a method to optimize sequential data annotation for off-policy evaluation (OPE) in offline reinforcement learning (RL). The core problem addressed is how to best utilize a limited budget for ground-truth data annotation when dealing with complex state and reward information, such as text or images, which are common in recent AI applications. Traditional methods often rely on expensive expert annotations or potentially biased outputs from tools like LLM-as-a-judge. The proposed approach optimizes annotation probabilities to achieve variance-optimal sequential OPE, where the target policy value is estimated from the annotated data. The researchers characterize these optimal annotation probabilities for sequential forward-monotone annotation protocols and provide a feasible batch-adaptive implementation. This work is motivated by real-world challenges, including a collaboration with a homelessness services nonprofit that uses casenotes to track client progress over time.

Why it matters for builders

AI builders working with offline RL and complex observational data can leverage this method to improve the reliability of their evaluations without requiring exhaustive ground-truth labeling. It provides a principled way to balance the cost of expert annotation against the potential biases of automated labeling, leading to more trustworthy inference from data that might otherwise be difficult to interpret accurately. This is particularly relevant for applications in social services, healthcare, e-commerce, and LLM training, where longitudinal interaction data is prevalent.

Practical impact

The research demonstrates significant reductions in Root Mean Squared Error (RMSE) for policy-value estimates on real datasets. For instance, in simulations and on two real datasets—casenotes from a nonprofit and human-preference votes from LMArena—the method achieved RMSE reductions of 34-65% for housing placement outcomes and 17-68% for progress towards a housing application, at annotation budgets of 40% of full annotation and above. On the LMArena dataset, reductions ranged from 55-62% across all tested budgets. These results suggest that builders can achieve substantial improvements in evaluation accuracy by strategically allocating their annotation resources, even when only a fraction of the data is expertly annotated. The method is designed to be batch-adaptive, facilitating practical implementation.

Caveats and source limits

The research is presented as a preprint on arXiv and has not undergone peer review. The specific implementation details for the batch-adaptive protocol are not fully elaborated in the provided excerpt, and further details would be necessary for direct application. While the paper discusses optimization for sequential annotation protocols, the full complexity of accounting for state impacts on Q-function estimation is noted as an approximation. The performance gains are described as smaller for very small pilot budgets, where the model has fewer labels to learn from, but persist as the budget increases.