aiqbench analysis
This research addresses the computational cost of evaluating classifiers in online settings by using Bayesian multi-armed bandits to adaptively select agents that balance predictive accuracy with subgroup fairness. By incorporating Thompson Sampling and handling delayed label observations, the study provides theoretical guarantees and empirical reductions in regret across tabular models, LLMs, and human labelers. The work advances the intersection of machine learning theory and algorithmic fairness by optimizing resource allocation without sacrificing ethical constraints.
Bayesian Learning to Select Fair Classifiers
arXiv:2610.20514v1 Announce Type: new Abstract: Evaluations of classifiers (AI or human or agents) often assume a batch setting in which every agent produces labels on every example in a test set, which can be expensive or impractical in many contexts. This motivates online policies that adaptively select agents to query as examples arrive and update policies as outcomes are observed. We study the problem of selecting, in an online stream, the agent that best balances (i) predictive performance and (ii) subgroup fairness of a sensitive attribute, and that allows for delayed observations of true class labels. We formulate this as a Bayesian multi-armed bandit problem using Thompson Sampling, explicitly balancing accuracy and fairness. We conduct empirical evaluations on real-world datasets spanning problems involving tabular classifiers, large language models, and human labelers, demonstrating substantial reductions in regret relative to baselines. Our theoretical contributions include guarantees on the performance of the selected agent that incorporate both the accuracy and fairness aspects of the problem.
Abstract:Evaluations of classifiers (AI or human or agents) often assume a batch setting in which every agent produces labels on every example in a test set, which can be expensive or impractical in many contexts. This motivates online policies that adaptively select agents to query as examples arrive and update policies as outcomes are observed. We study the problem of selecting, in an online stream, the agent that best balances (i) predictive performance and (ii) subgroup fairness of a sensitive attribute, and that allows for delayed observations of true class labels. We formulate this as a Bayesian multi-armed bandit problem using Thompson Sampling, explicitly balancing accuracy and fairness. We conduct empirical evaluations on real-world datasets spanning problems involving tabular classifiers, large language models, and human labelers, demonstrating substantial reductions in regret relative to baselines. Our theoretical contributions include guarantees on the performance of the selected agent that incorporate both the accuracy and fairness aspects of the problem.
Machine Learning (cs.LG)
arXiv:2610.20514 [cs.LG]
(or arXiv:2610.20514v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.20514 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 251 words
Related coverage
The Entangled Dimensions of Decision Theory
LessWrong · Jul 30, 2026
Base Models Can Detect Hidden Intentions that Prompted Classifiers Miss
LessWrong · Oct 12, 2026
Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face
AI Alignment Forum · Aug 3, 2026
[AINews] The Future of Latent Space
Latent Space · Sep 25, 2026