analiza aiqbench
This research introduces internal monitoring mechanisms based on the functional roles of specific model modules to detect training instability in large language models before loss divergence occurs. By analyzing spectral entropy in flash attention and router behavior for MoE architectures, these monitors can preemptively identify faults thousands of steps earlier than traditional metrics like gradient norms allow. Such early detection is critical for reducing wasted compute resources on frontier AI infrastructure where failure costs are substantial.
Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instability
arXiv:2606.28116v2 Announce Type: replace Abstract: Frontier large language model training consumes massive accelerator fleets and long wall-clock computation, making stability failures costly when they occur. After a numerical or a hyperparameter fault has already destabilized the training dynamics, it may continue for thousands of steps while loss and gradient norms still appear normal. We study mechanism-driven detection of training instability by deriving internal monitors from the functional role of each critical module and from the earliest computational sites where failures are expected to produce measurable signatures. For low-precision flash attention, we monitor the spectral entropy of a QK bilinear decomposition, whose first-order term becomes abnormal before the loss fully collapses. For MoE routers, we derive indicators from their role in expert selection. Our fault-injection experiments on low-precision attention, large learning-rate, and combined faults show that these signals provide distinct signatures for different failures, triggering thousands of steps before loss divergence.
Abstract:Frontier large language model training consumes massive accelerator fleets and long wall-clock computation, making stability failures costly when they occur. After a numerical or a hyperparameter fault has already destabilized the training dynamics, it may continue for thousands of steps while loss and gradient norms still appear normal. We study mechanism-driven detection of training instability by deriving internal monitors from the functional role of each critical module and from the earliest computational sites where failures are expected to produce measurable signatures. For low-precision flash attention, we monitor the spectral entropy of a QK bilinear decomposition, whose first-order term becomes abnormal before the loss fully collapses. For MoE routers, we derive indicators from their role in expert selection. Our fault-injection experiments on low-precision attention, large learning-rate, and combined faults show that these signals provide distinct signatures for different failures, triggering thousands of steps before loss divergence.
Computation and Language (cs.CL)
arXiv:2606.28116 [cs.CL]
(or arXiv:2606.28116v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.28116 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 238 cuvinte
Subiecte conexe
News
Quantum Computing Report · 7 iun. 2026
The art and science of hyperparameter optimization on Amazon Nova Forge
AWS · 2 iun. 2026
Against Export Controls (and China Threat Models)
LessWrong · 24 sept. 2026
The J-Space Debate, Agent Swarms, and Pacing Frontier AI - Digital Minds Newsletter #4
LessWrong · 18 sept. 2026