From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routing

Abstract:Existing LLM routing methods often construct supervision from a single sampled response for each query--model pair. Because LLM generation is stochastic, however, such an observation can be an unstable estimate of model capability: semantically equivalent query formulations and repeated decoding may yield different scores and even different model preferences. We show that this instability can further propagate from routing labels to learned routing policies. To address this issue, we propose DARS (Distribution-Aware Routing Supervision), which estimates query-level model capability from repeated observations spanning semantics-preserving query rewrites and stochastic decoding. DARS summarizes expected quality, expected cost, and performance variability to construct risk-aware supervision without changing the downstream router architecture. Experiments across diverse tasks and routing methods show that DARS generally improves routing utility and cost--quality trade-offs over single-shot supervision. Further analyses show that its benefits persist under moderate sampling budgets and different decoding temperatures. These results suggest that reliable LLM routing should move beyond individual sampled outcomes and instead model query-level capability distributions.

Machine Learning (cs.LG)

arXiv:2606.06924 [cs.LG]

(or arXiv:2606.06924v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2606.06924 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.