aiqbench analysis
This paper provides a rigorous theoretical foundation for decentralized autoregressive generation by proving its equivalence to centralized training using Discrete Flow Matching. The authors demonstrate that global models can naturally decompose into independent experts without sacrificing performance on multimodal benchmarks, addressing key scaling bottlenecks in the field. This work bridges the gap between empirical success and formal justification for distributed AI architectures.
Decentralized Autoregressive Generation
arXiv:2601.03184v3 Announce Type: replace-cross Abstract: The decentralization of autoregressive generation has attracted considerable attention in recent years as a solution to scaling bottlenecks. However, despite promising empirical results, this paradigm currently lacks rigorous theoretical justification. In this work, we formally establish the theoretical equivalence between decentralized and centralized training. To achieve this, we adapt the Discrete Flow Matching framework for autoregressive generation, leveraging its inherent properties to demonstrate that global models naturally decompose into independent experts. Finally, we conduct extensive experiments across diverse multimodal benchmarks, empirically validating that decentralized training maintains competitive parity with standard centralized architectures.
Abstract:The decentralization of autoregressive generation has attracted considerable attention in recent years as a solution to scaling bottlenecks. However, despite promising empirical results, this paradigm currently lacks rigorous theoretical justification. In this work, we formally establish the theoretical equivalence between decentralized and centralized training. To achieve this, we adapt the Discrete Flow Matching framework for autoregressive generation, leveraging its inherent properties to demonstrate that global models naturally decompose into independent experts. Finally, we conduct extensive experiments across diverse multimodal benchmarks, empirically validating that decentralized training maintains competitive parity with standard centralized architectures.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2601.03184 [cs.LG]
(or arXiv:2601.03184v3 [cs.LG] for this version)
Full text under the publisher's open licence · 106 words