aiqbench analysis
This paper introduces the "2-Step Agent" framework to model how rational decision-makers update beliefs based on ML predictions, revealing that even well-specified models can degrade outcomes if user priors are misaligned. The authors provide a tractable solution for Bayesian inference in linear Gaussian settings and empirically identify specific conditions where ML decision support is beneficial versus harmful. Ultimately, the work underscores the critical importance of user priors and belief calibration in high-stakes domains like healthcare and the judiciary, suggesting that decision support systems can actively cause harm under ideal model conditions alone.
2-Step Agent: A Framework for the Interaction of a Decision Maker with AI Decision Support
arXiv:2602.21889v2 Announce Type: replace Abstract: Predictions from ML models support human decision making in several fields, including high-stakes ones such as healthcare and the judiciary. Yet, we still lack a clear understanding of how decision makers learn from ML-based decision support (ML-DS). In this paper, we introduce a general computational framework, the 2-Step Agent, to capture this process. As a prediction from an ML model contains information about the training data, a prediction can also be used for inference. Our framework models (i) how a prediction for a new observation affects the beliefs of a rational Bayesian agent, and (ii) how this change in beliefs affects the estimation of causal effect, the downstream decision, and the subsequent outcome. In addition to the framework itself, we make three contributions. First, for the linear Gaussian setting, we derive a tractable solution for the challenging Bayesian inference problem we introduced, i.e. one in which the agent infers from an ML prediction. Second, we experimentally identify conditions under which ML-DS is beneficial. Third, we show that a single misaligned prior belief can be sufficient for ML-DS to lead to worse downstream outcomes compared to no decision support even when the ML model is well-specified and the agent is perfectly rational. Hence, even under ideal conditions, ML-DS can do more harm than good. % if users have incorrect beliefs about the ML
Abstract:Predictions from ML models support human decision making in several fields, including high-stakes ones such as healthcare and the judiciary. Yet, we still lack a clear understanding of how decision makers learn from ML-based decision support (ML-DS). In this paper, we introduce a general computational framework, the 2-Step Agent, to capture this process. As a prediction from an ML model contains information about the training data, a prediction can also be used for inference. Our framework models (i) how a prediction for a new observation affects the beliefs of a rational Bayesian agent, and (ii) how this change in beliefs affects the estimation of causal effect, the downstream decision, and the subsequent outcome. In addition to the framework itself, we make three contributions. First, for the linear Gaussian setting, we derive a tractable solution for the challenging Bayesian inference problem we introduced, i.e. one in which the agent infers from an ML prediction. Second, we experimentally identify conditions under which ML-DS is beneficial. Third, we show that a single misaligned prior belief can be sufficient for ML-DS to lead to worse downstream outcomes compared to no decision support even when the ML model is well-specified and the agent is perfectly rational. Hence, even under ideal conditions, ML-DS can do more harm than good. % if users have incorrect beliefs about the ML
Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2602.21889 [cs.AI]
(or arXiv:2602.21889v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2602.21889 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 317 words
Related coverage
Hint-based CoT faithfulness evals still mostly work on Claude
LessWrong · Jul 30, 2026
How Sonos rebooted itself
The Verge · Sep 3, 2026
Underwriting Superintelligence: Backing Agents you can Sue — Rune Kvist, AIUC
Latent Space · Sep 16, 2026
2-Step Agent: A Framework for the Interaction of a Decision Maker with AI Decision Support
arXiv · Jun 9, 2026