Mixture-of-Models: Distributed Inference over Heterogeneous LLMs as Model-Level Mixture-of-Experts

Abstract:Switch Transformers scale MoE model capacity by routing tokens in memory for one jointly trained model, requiring high-bandwidth interconnects, large memory, centralised ownership, and lack scalable composability. We propose Mixture-of-Models (MoM) architecture as Switch Transformer scaling to semantic level which composes inference across independently trained distributed LLMs, each run on its own GPU. The models reach joint consensus using N-Way Self-Evaluating Deliberation protocol: a combination of social-choice methods such as Delphi method and Quadratic Voting as well as Inference-Time scaling paradigm. We start with an empirical model that characterizes consensus as an inference-time trade-off between signal extraction and contextual noise accumulation, establishing a mathematical basis for convergence driven inference stopping strategies. We show experimental results where ensemble of smaller (<21B) models matches closely 120B MoE (GPT-OSS-120B) on AIME 2025 while larger ensemble of open-weight models reaches 90.0%, close to proprietary models at time of writing.

Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multiagent Systems (cs.MA); Systems and Control (eess.SY)

arXiv:2601.16863 [cs.AI]

(or arXiv:2601.16863v2 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2601.16863 Focus to learn more arXiv-issued DOI via DataCite

https://doi.org/10.5281/zenodo.18234923 Focus to learn more DOI(s) linking to related resources

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.