aiqbench analysis
The study identifies a fundamental "one-bit wall" in sub-bit model compression, where weight signs remain locked to their initialization values and resist low-rank approximation. This sign lock-in phenomenon suggests that achieving storage below one bit per weight is inherently limited by the persistence of initialization patterns rather than learned information. Researchers can bypass this bottleneck by adopting a specific low-rank sign-template training method that prevents the emergence of this fixed-cost bottleneck.
Sign Lock-In: Randomly Initialized Weight Signs Persist and Bottleneck Sub-Bit Model Compression
arXiv:2602.17063v2 Announce Type: replace-cross Abstract: Sub-bit model compression targets storage below one bit per weight; as magnitudes are aggressively compressed, the sign bit becomes a fixed-cost bottleneck. Across Transformers, CNNs, and MLPs, learned sign matrices resist low-rank approximation and are spectrally indistinguishable from an i.i.d. Rademacher baseline. This randomness gives rise to the lower bound of sub-bit model compression -- the one-bit wall. Despite this apparent randomness, most weights retain their initialization signs; flips primarily occur via rare near-zero boundary crossings, suggesting that sign-pattern randomness is largely inherited from initialization. We formalize this behavior with sign lock-in theory, a stopping-time analysis of sign flips under SGD noise. Under bounded updates and a rare re-entry condition into a small neighborhood of zero, the number of effective sign flips exhibits a geometric tail. Building on this mechanism, we introduce a from-scratch low-rank sign-template training method that prevents the emergence of this one-bit wall.
Abstract:Sub-bit model compression targets storage below one bit per weight; as magnitudes are aggressively compressed, the sign bit becomes a fixed-cost bottleneck. Across Transformers, CNNs, and MLPs, learned sign matrices resist low-rank approximation and are spectrally indistinguishable from an i.i.d. Rademacher baseline. This randomness gives rise to the lower bound of sub-bit model compression -- the one-bit wall. Despite this apparent randomness, most weights retain their initialization signs; flips primarily occur via rare near-zero boundary crossings, suggesting that sign-pattern randomness is largely inherited from initialization. We formalize this behavior with sign lock-in theory, a stopping-time analysis of sign flips under SGD noise. Under bounded updates and a rare re-entry condition into a small neighborhood of zero, the number of effective sign flips exhibits a geometric tail. Building on this mechanism, we introduce a from-scratch low-rank sign-template training method that prevents the emergence of this one-bit wall.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
arXiv:2602.17063 [cs.LG]
(or arXiv:2602.17063v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2602.17063 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 251 words
Related coverage
Beyond Retraining-Free MoE Compression: A Cost-Normalized Study of Post-Compression Adjustment
arXiv · Sep 9, 2026
Just Initialize: A Training-Free Initialization Component for Large-Scale Routing Optimization
arXiv · Sep 29, 2026
Towards Reliable Context Compression for Long-Horizon Agents: An Empirical Study of Execution Instability
arXiv · Oct 16, 2026
TRACER: Per-Tool Context Retention for LLM Agents via Consequence-Attributed Reinforcement Learning
arXiv · Sep 1, 2026