analiza aiqbench
MoP addresses the computational bottleneck in Large Language Models by unifying depth and width pruning into a single iterative framework that consistently outperforms existing methods on LLaMA-2/3 architectures. By achieving significant latency reductions without sacrificing accuracy, this approach offers a practical pathway for deploying efficient models across diverse applications including vision-language tasks like LLaVA.
Compressing LLMs with MoP: Mixture of Pruners
arXiv:2602.06127v2 Announce Type: replace Abstract: The high computational demands of Large Language Models (LLMs) motivate methods that reduce parameter count and accelerate inference. In response, model pruning emerges as an effective strategy, yet current methods typically focus on a single dimension-depth or width. We introduce MoP (Mixture of Pruners), an iterative framework that unifies these dimensions. At each iteration, MoP generates two branches-pruning in depth versus pruning in width-and selects a candidate to advance the path. On LLaMA-2 and LLaMA-3, MoP advances the frontier of structured pruning, exceeding the accuracy of competing methods across a broad set of compression regimes. It also consistently outperforms depth-only and width-only pruning. Furthermore, MoP translates structural pruning into real speedup, reducing end-to-end latency by 39% at 40% compression. Finally, extending MoP to the vision-language model LLaVA-1.5, we notably improve computational efficiency and demonstrate that text-only recovery fine-tuning can restore performance even on visual tasks.
Abstract:The high computational demands of Large Language Models (LLMs) motivate methods that reduce parameter count and accelerate inference. In response, model pruning emerges as an effective strategy, yet current methods typically focus on a single dimension-depth or width. We introduce MoP (Mixture of Pruners), an iterative framework that unifies these dimensions. At each iteration, MoP generates two branches-pruning in depth versus pruning in width-and selects a candidate to advance the path. On LLaMA-2 and LLaMA-3, MoP advances the frontier of structured pruning, exceeding the accuracy of competing methods across a broad set of compression regimes. It also consistently outperforms depth-only and width-only pruning. Furthermore, MoP translates structural pruning into real speedup, reducing end-to-end latency by 39% at 40% compression. Finally, extending MoP to the vision-language model LLaVA-1.5, we notably improve computational efficiency and demonstrate that text-only recovery fine-tuning can restore performance even on visual tasks.
Machine Learning (cs.LG)
arXiv:2602.06127 [cs.LG]
(or arXiv:2602.06127v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2602.06127 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 237 cuvinte
Subiecte conexe
What's Continual Learning, and Why Might We Expect To See It In Advanced LLM Agents?
LessWrong · 12 iun. 2026
Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs
arXiv · 25 aug. 2026
Pruning Laws for Large Language Models
arXiv · 31 aug. 2026
OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit
arXiv · 29 sept. 2026