aiqbench analysis
Routing LLM queries to different models only pays off when a learned performance predictor beats a simple baseline: the model's own confidence in its answer. The paper shows that such gains are bounded by how suboptimal the candidate model is, plus the extra capacity the predictor adds, and that the advantage shrinks as models are finetuned. For readers building cost-aware routing or cascades, this suggests that predictor overhead is hard to justify for strong models, and that calibrated self-confidence is a stronger baseline than it may look.
Predictable performance is suboptimal, and what it means for model routing
arXiv:2610.31428v1 Announce Type: new Abstract: Large language models (LLMs) differ in capabilities and costs, motivating routing methods that adaptively choose a model for each query. These methods rely on performance prediction: given a query, estimate whether a model will answer it correctly. Most routing methods train separate predictors of candidate models' performance to guide model selection. Our work shows that performance prediction only beats a simple baseline when the candidate model is suboptimal on the task or the performance predictor adds significant capacity relative to the candidate model. Specifically, we take a model's self-confidence, the probability it assigns to its chosen answer, as a baseline and study how much a performance predictor can improve on it. First, we show that any performance predictor that outperforms a model's self-confidence can be used to improve the model's loss: the predictor's advantage transfers exactly in log loss through recalibration. Second, accounting for the predictor's additional parameters, we bound its advantage by the model's suboptimality plus the gain from those parameters. Third, we extend our theoretical results to two-model cascades, which route queries to a more capable model when the first model is predicted to fail. Experiments with 16 language models on four benchmarks show that predictors gain a larger advantage over self-confidence on more suboptimal models. This advantage shrinks as models improve through finetuning. Cascading exhibits the same two patterns: predictor-based routing offers larger gains over confidence-based routing for more suboptimal models, and these gains diminish after finetuning.
Abstract:Large language models (LLMs) differ in capabilities and costs, motivating routing methods that adaptively choose a model for each query. These methods rely on performance prediction: given a query, estimate whether a model will answer it correctly. Most routing methods train separate predictors of candidate models' performance to guide model selection. Our work shows that performance prediction only beats a simple baseline when the candidate model is suboptimal on the task or the performance predictor adds significant capacity relative to the candidate model. Specifically, we take a model's self-confidence, the probability it assigns to its chosen answer, as a baseline and study how much a performance predictor can improve on it. First, we show that any performance predictor that outperforms a model's self-confidence can be used to improve the model's loss: the predictor's advantage transfers exactly in log loss through recalibration. Second, accounting for the predictor's additional parameters, we bound its advantage by the model's suboptimality plus the gain from those parameters. Third, we extend our theoretical results to two-model cascades, which route queries to a more capable model when the first model is predicted to fail. Experiments with 16 language models on four benchmarks show that predictors gain a larger advantage over self-confidence on more suboptimal models. This advantage shrinks as models improve through finetuning. Cascading exhibits the same two patterns: predictor-based routing offers larger gains over confidence-based routing for more suboptimal models, and these gains diminish after finetuning.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.31428 [cs.LG]
(or arXiv:2610.31428v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.31428 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 337 words
Related coverage
Temporal Predictive Multiplicity: Equally Accurate Time Series Models Yield Different Forecast Trajectories
arXiv · Oct 8, 2026
Partially Performative Prediction
arXiv · Jun 9, 2026
A panoramic aerodynamic performance prediction method for turbomachinery cascades using transformer-enhanced neural operator
arXiv · Sep 16, 2026
Beyond Point Prediction: Artificial Representative Trees with Uncertainty
arXiv · Sep 22, 2026