The Behavioral Credibility Trilemma: When Calibrated Autonomy Becomes Impossible

Abstract:We prove that no reinforcement learning policy with confidence-gated autonomy can simultaneously achieve maximum helpfulness, optimal calibration, and full autonomy under rational oversight, whenever some tasks exceed the agent's reliable competence: the Behavioral Credibility Trilemma. The impossibility is geometric: adding any non-affine autonomy incentive to a strictly proper scoring rule destroys strict properness, so an agent rewarded for both calibrated confidence and autonomous action systematically inflates its reported confidence on tasks below the principal's approval threshold whenever the autonomy stake exceeds the calibration cost of clearing it. The Behavioral Perturbation Lemma quantifies the inflation (scaling as $w_A/(2 w_C)$ for the Brier score) and shows detection requires $\Omega(1/\Delta^2)$ observations for interior reports. We prove that, in the unsaturated regime, no affine oversight rule is optimal for the principal and the optimum is attained by a sharp threshold satisfying the trilemma's own hypotheses, so the impossibility is endogenized rather than assumed; moreover, for symmetric, log-concave, full-support location policy families under the Brier score, calibration is not even a stationary point of policy-gradient training. We formalize the Confidence-Gated Decision Problem, map existing methods onto the trilemma, and identify two constructive resolution pathways (commitment, role separation). A 540-configuration Best-of-N experiment tests five hypotheses, all strongly confirmed (effect sizes $d = 1.10$ to $5.35$, the upper end from a per-completion estimator inflating magnitude over per-task aggregates) and replicated under a pre-specified protocol on two further model families, and adds a descriptive analysis of the achievable-$(H, C, A)$ surface geometry showing a plateau-truncated frontier consistent with the predicted inflation saturation.

Machine Learning (cs.LG); Computer Science and Game Theory (cs.GT); Machine Learning (stat.ML)

arXiv:2605.25739 [cs.LG]

(or arXiv:2605.25739v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2605.25739 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.