analiza aiqbench
This research introduces a polynomial preconditioning layer that stabilizes weight conditioning during LLM pre-training without adding inference overhead, as the parameters can be merged back into the standard architecture. Theoretically grounded in singular-value spectrum control, the method demonstrates improved convergence for both AdamW and Muon optimizers on Llama-1B models. The approach offers a practical pathway to enhance training stability while maintaining computational efficiency for existing transformer architectures.
PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training
arXiv:2606.06470v1 Announce Type: cross Abstract: We propose a preconditioning (PC) layer, a weight parameterization via polynomial preconditioner that ensures stable weight conditioning throughout LLM training. The PC module reshapes the singular-value spectrum of weight matrices via low-degree polynomial preconditioning. After training, the preconditioned weights can be merged back into the original architecture, incurring no inference overhead. We demonstrate the advantage of the proposed PC layer over standard transformers in Llama-1B pre-training, for both the AdamW and Muon optimizers. Theoretically, we justify this spectrum-control principle by proving that uniformly bounding each layer's singular values ensures geometric convergence of gradient descent to global minima, for certain deep linear networks. Our code is available at https://github.com/Empath-aln/PC-layer.
Abstract:We propose a preconditioning (PC) layer, a weight parameterization via polynomial preconditioner that ensures stable weight conditioning throughout LLM training. The PC module reshapes the singular-value spectrum of weight matrices via low-degree polynomial preconditioning. After training, the preconditioned weights can be merged back into the original architecture, incurring no inference overhead. We demonstrate the advantage of the proposed PC layer over standard transformers in Llama-1B pre-training, for both the AdamW and Muon optimizers. Theoretically, we justify this spectrum-control principle by proving that uniformly bounding each layer's singular values ensures geometric convergence of gradient descent to global minima, for certain deep linear networks. Our code is available at this https URL.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2606.06470 [cs.LG]
(or arXiv:2606.06470v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.06470 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 207 cuvinte
Subiecte conexe
A Mike's-Eye View of ARC's Research
AI Alignment Forum · 9 iun. 2026
Generalization and infinite width
LessWrong · 1 aug. 2026
Advancing Emerging Optimizers for Accelerated LLM Training with NVIDIA Megatron
NVIDIA · 22 apr. 2026
Activation-Aware Weight Tensorization: A Calibration-Time Preconditioner for Tensor-Network LLM Compression
arXiv · 8 oct. 2026