analiza aiqbench
This research introduces nGPT, a normalized Transformer architecture that constrains model parameters and activations to the unit hypersphere to improve training efficiency on hybrid Mamba-2--Transformer MoE models. By implementing techniques like Logit Gradient Preconditioning and angular update control, the approach achieves comparable validation loss using approximately half the number of tokens required by standard unnormalized baselines. The study demonstrates significant token reduction across scales up to 14B parameters while maintaining architectural flexibility for modern hybrid systems.
Training nGPT
arXiv:2608.01284v1 Announce Type: cross Abstract: The normalized Transformer (nGPT) realizes hyperspherical representation learning by constraining model parameter vectors and activation vectors to the unit hypersphere. In this paper, we describe a practical training recipe for nGPT and evaluate it on modern hybrid Mamba-2--Transformer Mixture-of-Experts (MoE) models. The recipe introduces Logit Gradient Preconditioning, Logarithmic Learning Rate Decay, GatedAdamW, angular update control, and optional exploration mechanisms. Compared with an unnormalized model of the same hybrid MoE architecture trained with AdamW, the 14B-total-parameter nGPT model reaches the same validation loss using approximately half as many training tokens. The recipe scales across the models considered, which contain up to 14B total parameters.
Abstract:The normalized Transformer (nGPT) realizes hyperspherical representation learning by constraining model parameter vectors and activation vectors to the unit hypersphere. In this paper, we describe a practical training recipe for nGPT and evaluate it on modern hybrid Mamba-2--Transformer Mixture-of-Experts (MoE) models. The recipe introduces Logit Gradient Preconditioning, Logarithmic Learning Rate Decay, GatedAdamW, angular update control, and optional exploration mechanisms. Compared with an unnormalized model of the same hybrid MoE architecture trained with AdamW, the 14B-total-parameter nGPT model reaches the same validation loss using approximately half as many training tokens. The recipe scales across the models considered, which contain up to 14B total parameters.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2608.01284 [cs.LG]
(or arXiv:2608.01284v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2608.01284 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 200 cuvinte
Subiecte conexe
You can graft SDF changes from base models onto post-trained models
LessWrong · 6 oct. 2026
Scale Robot Reinforcement Learning with NVIDIA Isaac Lab on Amazon SageMaker AI
AWS · 9 iun. 2026
Introducing Instella-MoE: A State-of-the-Art Fully Open Mixture-of-Experts Language Model
AMD · 24 iul. 2026
Towards embedded evaluations for scheming propensities
LessWrong · 8 oct. 2026