analiza aiqbench
This research addresses parameter redundancy in large language models by introducing CS-MoE, an architecture that shares experts across transformer layers to reduce computational costs. The approach achieves lower perplexity than dense models while activating only 55% of parameters, establishing a flexible trade-off between efficiency and capacity. This offers a viable path for deploying high-performance models in resource-constrained environments without sacrificing accuracy.
Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformers
arXiv:2609.22199v1 Announce Type: new Abstract: Transformer-based large language models often suffer from inter-layer parameter redundancy, where functional transformations are redundantly learned across network depths. We propose CS-MoE, a novel Transformer architecture featuring cross-layer expert sharing to address this inefficiency. Deviating from the widely used Mixture-of-Experts (MoE) architecture that terminates each Transformer block with layer-isolated experts, CS-MoE combines layer-independent experts with concurrent access to a centralized, globally shared expert pool. This \textit{Global Experts Sharing} mechanism enables elastic control over token-level parameter activation and computational consumption (FLOPs). Experiments demonstrate that CS-MoE achieves lower perplexity than equal-scale dense Transformers while activating only 55\% of parameters. Furthermore, its performance scales monotonically with an increased number of activated experts and approaches MoE counterparts that consume more FLOPs by expanding the shared pool with a fixed FLOPs budget. CS-MoE also establishes a flexible Pareto frontier between computational cost and model capacity, offering an efficient alternative for computation-constrained environments.
Abstract:Transformer-based large language models often suffer from inter-layer parameter redundancy, where functional transformations are redundantly learned across network depths. We propose CS-MoE, a novel Transformer architecture featuring cross-layer expert sharing to address this inefficiency. Deviating from the widely used Mixture-of-Experts (MoE) architecture that terminates each Transformer block with layer-isolated experts, CS-MoE combines layer-independent experts with concurrent access to a centralized, globally shared expert pool. This \textit{Global Experts Sharing} mechanism enables elastic control over token-level parameter activation and computational consumption (FLOPs). Experiments demonstrate that CS-MoE achieves lower perplexity than equal-scale dense Transformers while activating only 55\% of parameters. Furthermore, its performance scales monotonically with an increased number of activated experts and approaches MoE counterparts that consume more FLOPs by expanding the shared pool with a fixed FLOPs budget. CS-MoE also establishes a flexible Pareto frontier between computational cost and model capacity, offering an efficient alternative for computation-constrained environments.
11 pages, 6 figures. To appear in the Proceedings of EMNLP 2026 Main
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2609.22199 [cs.LG]
(or arXiv:2609.22199v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.22199 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 255 cuvinte
Subiecte conexe
The hard physics and complex economics of AI’s insatiable hunger for power
SiliconANGLE · 11 sept. 2026
[AINews] not much happened today
Latent Space · 10 sept. 2026
[AINews] not much happened today
Latent Space · 3 oct. 2026
[AINews] Agents disprove Polysemanticity; greatly improve Interpretability & Alignment
Latent Space · 16 oct. 2026