Batch-Adaptive Causal Annotations: An Application to Homelessness Street Outreach Casenotes

Estimating the causal effects of an intervention on outcomes is crucial. But often in domains such as healthcare and social services, this critical information about outcomes is documented by unstructured text, e.g. clinical notes in healthcare or case notes in social services. We are motivated by an ongoing collaboration with a nonprofit providing street outreach to unhoused clients. Outreach is a common social services intervention, with ambiguous and hard-to-measure outcomes. Outreach workers write case notes after every interaction, recording notable events and client progress beyond final housing placement outcomes. Although experts can succinctly extract relevant information from such unstructured case notes, it is costly or simply infeasible for an entire corpus (millions of notes). Recent advances in large language models (LLMs) enable scalable but potentially inaccurate annotation of unstructured text data. Under a framework of causal inference with missing outcomes, we develop estimators that robustly combine expert annotation vs. noisy imputation. To best leverage a limited annotation budget, we develop a two-stage adaptive algorithm that optimizes the expert annotation probabilities, estimating the average treatment effect with optimal asymptotic variance. Expert labels and LLM annotations can be combined efficiently and responsibly in causal estimation. More generally, our work discusses the optimal choice of validation data for causal inference with non-standard measurement error. We discuss extensions to more complex estimands. This is joint work with Ezinne Nwankwo and Lauri Goldkind.

Angela Zhou is an assistant professor at USC Marshall Data Sciences and Operations, in the Operations group, and in the Department of Computer Science (by courtesy). She works on applications-motivated theoretically-guaranteed methodology in statistical machine learning, operations management, and causal inference, including on algorithmic fairness. Her most recent work in AI and society focuses on the operational service provision of safety-net social services.