aiqbench analysis
This research addresses a critical challenge in applied AI by optimizing how limited human budgets are allocated for labeling data used with Large Language Models (LLMs) to estimate causal effects. The study demonstrates that strategic selection of annotations significantly reduces costs while correcting inherent biases where out-of-the-box LLMs fail to accurately recognize client progress in social services casenotes. For practitioners deploying AI in policy or operations, this offers a rigorous framework for maintaining estimation validity when ground-truth data is scarce and expensive to obtain.
Optimal Causal Annotations: An Application to Casenotes in Social Services
arXiv:2502.10605v4 Announce Type: replace-cross Abstract: Problem definition: Estimating causal effects of interventions is central to policy and operations, but outcome data are often missing or costly to obtain. LLMs can provide text annotation at scale but may be subject to unknown bias. When ground-truth outcomes require expensive expert labeling or follow-up, budget limits typically allow only a fraction of the data to be labeled. Motivated by collaboration with a nonprofit conducting street outreach in homelessness services, whose most interesting outcomes are in unstructured casenotes, we ask: which observations should be selected for labeling under a fixed budget? Methodology/results: Our method optimizes annotation probabilities to minimize the variance of average treatment effect estimation. We derive a closed-form solution and establish that a feasible two-batch estimator achieves the best possible asymptotic variance. On simulated and real-world datasets, our method achieves lower MSE than random sampling and 43%-91% reductions in labels and costs for the same interval widths. In a case study classifying progress towards housing applications from casenotes with LLMs, we find out-of-the-box LLMs under-recognize client progress, highlighting the importance of grounding LLM judgment. Our estimates indicate that 8.6% of clients improve 2-year housing outcomes due to more street outreach in the first six months, and increasing outreach from the first to third quartile increases maximum progress towards a housing application by an estimated half-step. Managerial implications: LLM-as-a-judge may threaten estimation validity; our method enables valid causal estimation with a limited annotation budget. Evaluating causal impacts on intermediate nonprofit outcomes can inform operations: returns to outreach appear concave, so expanding the extensive margin of who receives outreach may be resource-efficient.
Abstract:Problem definition: Estimating causal effects of interventions is central to policy and operations, but outcome data are often missing or costly to obtain. LLMs can provide text annotation at scale but may be subject to unknown bias. When ground-truth outcomes require expensive expert labeling or follow-up, budget limits typically allow only a fraction of the data to be labeled. Motivated by collaboration with a nonprofit conducting street outreach in homelessness services, whose most interesting outcomes are in unstructured casenotes, we ask: which observations should be selected for labeling under a fixed budget? Methodology/results: Our method optimizes annotation probabilities to minimize the variance of average treatment effect estimation. We derive a closed-form solution and establish that a feasible two-batch estimator achieves the best possible asymptotic variance. On simulated and real-world datasets, our method achieves lower MSE than random sampling and 43%-91% reductions in labels and costs for the same interval widths. In a case study classifying progress towards housing applications from casenotes with LLMs, we find out-of-the-box LLMs under-recognize client progress, highlighting the importance of grounding LLM judgment. Our estimates indicate that 8.6% of clients improve 2-year housing outcomes due to more street outreach in the first six months, and increasing outreach from the first to third quartile increases maximum progress towards a housing application by an estimated half-step. Managerial implications: LLM-as-a-judge may threaten estimation validity; our method enables valid causal estimation with a limited annotation budget. Evaluating causal impacts on intermediate nonprofit outcomes can inform operations: returns to outreach appear concave, so expanding the extensive margin of who receives outreach may be resource-efficient.
Extended journal version. A preliminary version was accepted to AISTATS 2026
Machine Learning (stat.ML); Computers and Society (cs.CY); Machine Learning (cs.LG); Econometrics (econ.EM); Methodology (stat.ME)
arXiv:2502.10605 [stat.ML]
(or arXiv:2502.10605v4 [stat.ML] for this version)
Full text under the publisher's open licence · 296 words
Related coverage
Optimal Sequential Annotations for Off-Policy Evaluation
arXiv · Sep 23, 2026
Amazon Research Awards recipients announced
Amazon Science · May 27, 2026
Amazon is investing in the Lean Focused Research Organization
Amazon Science · Jul 26, 2026
34 Amazon Research Awards Build on Trainium recipients announced
Amazon Science · Aug 5, 2026