analiza aiqbench
AIR introduces a novel SVD-based compression framework for large language models that incorporates backward-signal influence metrics to optimize low-rank approximations. By integrating this approach with methods like LoRA, the technique achieves significant improvements in perplexity and efficiency while reducing calibration data requirements. This research advances the state of model quantization by directly translating parameter savings into tangible gains in FLOPs, memory usage, and latency.
Activation- and Influence-Aware Ranks (AIR): Function-Preserving SVD Compression for LLMs
arXiv:2606.19993v2 Announce Type: replace Abstract: We present Activation- and Influence-Aware Ranks (AIR), an SVD-based LLM compression framework that guides each weight matrix's low-rank approximation with a backward-signal influence metric. Starting from the activation-aware optimum of SVD-LLM(W), AIR runs a single closed-form alternating least squares (ALS) sweep that integrates influence element-wise under a monotone-descent guarantee. AIR is layer-local and composes orthogonally with end-to-end methods: alone it exceeds ACIP, and AIR+LoRA outperforms it further. AIR improves perplexity over SVD-LLM(W) by >18% at <=60% parameter retention, matches its quality with ~90% less calibration data, and turns parameter savings into FLOP, peak-memory, and per-token latency gains.
Abstract:We present Activation- and Influence-Aware Ranks (AIR), an SVD-based LLM compression framework that guides each weight matrix's low-rank approximation with a backward-signal influence metric. Starting from the activation-aware optimum of SVD-LLM(W), AIR runs a single closed-form alternating least squares (ALS) sweep that integrates influence element-wise under a monotone-descent guarantee. AIR is layer-local and composes orthogonally with end-to-end methods: alone it exceeds ACIP, and AIR+LoRA outperforms it further. AIR improves perplexity over SVD-LLM(W) by >18% at <=60% parameter retention, matches its quality with ~90% less calibration data, and turns parameter savings into FLOP, peak-memory, and per-token latency gains.
Machine Learning (cs.LG)
arXiv:2606.19993 [cs.LG]
(or arXiv:2606.19993v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.19993 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 189 cuvinte
Subiecte conexe
State of the Field: AI for Epistemics and Coordination
LessWrong · 2 oct. 2026
Reflections on unlearning and inoculation
LessWrong · 20 sept. 2026
Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Too
arXiv · 14 oct. 2026
Game Design Creativity evals
LessWrong · 12 oct. 2026