analiza aiqbench
The Loopie series introduces Mixture-of-Experts models utilizing looped transformer architectures, directly addressing the computational inefficiency typically associated with this design pattern compared to scaling parameter counts linearly. By combining novel post-training methods with extensive ablation studies against vanilla baselines, these models demonstrate that looping can achieve frontier-level reasoning performance without sacrificing efficiency under fixed compute budgets. This research advances architectural optimization for large language models by proving that structural repetition in transformers can yield superior capabilities when paired with specific training strategies.
Loop the Loopies!
arXiv:2607.16051v2 Announce Type: replace-cross Abstract: We present the Loopie series, consisting of two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6B-parameter model with 0.6B active parameters. Looped Transformers have long faced a challenge: given an N times increase in pre-training compute, increasing the parameter count by a factor of N usually outperforms looping a model N times. Loopie addresses this challenge. Extensive ablation studies, including comparisons with a vanilla 30B-A3B model, show that Loopie substantially outperforms vanilla Transformer baselines trained with the same compute budget. With a novel post-training method, Loopie develops strong reasoning abilities and achieves frontier-level reasoning performance.
Title:Loop the Loopies!
Abstract:We present the Loopie series, consisting of two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6B-parameter model with 0.6B active parameters. Looped Transformers have long faced a challenge: given an N times increase in pre-training compute, increasing the parameter count by a factor of N usually outperforms looping a model N times. Loopie addresses this challenge. Extensive ablation studies, including comparisons with a vanilla 30B-A3B model, show that Loopie substantially outperforms vanilla Transformer baselines trained with the same compute budget. With a novel post-training method, Loopie develops strong reasoning abilities and achieves frontier-level reasoning performance.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2607.16051 [cs.CL]
(or arXiv:2607.16051v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2607.16051 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 199 cuvinte
