analiza aiqbench
This research introduces the TSVD framework to address the high computational costs of LLM pretraining by efficiently maintaining orthonormality in low-rank weight matrices. By leveraging QR decomposition and a spectral energy heuristic, the method achieves performance comparable to full-parameter baselines at a fraction of the compute cost. The findings offer a scalable pathway for reducing the resource intensity of training large language models without sacrificing model quality.
Efficient Pre-Training of LLMs through Truncated SVD Representations
arXiv:2605.28573v2 Announce Type: replace Abstract: LLM pretraining is extremely costly; therefore, parameter-efficient LLM architectures have recently emerged as a compelling research direction. One such promising approach is to represent the parameters as orthonormal low-rank weight matrices. However, maintaining orthonormality during training is computationally expensive, making it impractical. This paper presents the TSVD (Truncated Singular Value Decomposition) framework which efficiently maintains orthonormality through QR decomposition and caching. Furthermore, a spectral energy heuristic is introduced to select the rank of the resulting low-rank weight matrices. Empirical evaluations across model sizes show that TSVD matches or outperforms full-parameter baselines at a fraction of the compute cost. TSVD thus provides a scalable, computationally efficient foundation for LLM pretraining.
Abstract:LLM pretraining is extremely costly; therefore, parameter-efficient LLM architectures have recently emerged as a compelling research direction. One such promising approach is to represent the parameters as orthonormal low-rank weight matrices. However, maintaining orthonormality during training is computationally expensive, making it impractical. This paper presents the TSVD (Truncated Singular Value Decomposition) framework which efficiently maintains orthonormality through QR decomposition and caching. Furthermore, a spectral energy heuristic is introduced to select the rank of the resulting low-rank weight matrices. Empirical evaluations across model sizes show that TSVD matches or outperforms full-parameter baselines at a fraction of the compute cost. TSVD thus provides a scalable, computationally efficient foundation for LLM pretraining.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2605.28573 [cs.LG]
(or arXiv:2605.28573v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.28573 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 204 cuvinte
Subiecte conexe
Does DiffusionGemma do latent reasoning?
AI Alignment Forum · 16 aug. 2026
Why Dwarkesh is Wrong about Computer Use + How OpenAI shipped its Jev competitor in 1 Week
Latent Space · 30 sept. 2026
Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models
arXiv · 21 iul. 2026
Invariant Gradient Alignment for Robust Reasoning Distillation
arXiv · 5 iun. 2026