analiza aiqbench
LayerTrim introduces a novel, training-free approach to structured pruning that specifically preserves long-form reasoning capabilities in large language models by selectively trimming MLP features while maintaining attention mechanisms. This method significantly reduces inference costs and GPU memory usage without the performance collapse typically seen when compressing models optimized for complex tasks like mathematical problem solving. The technique represents a meaningful advancement in making high-reasoning LLMs more efficient for practical deployment.
Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming
arXiv:2610.19121v1 Announce Type: cross Abstract: Structured pruning reduces LLM inference costs by removing model components to produce smaller dense models. However, existing methods can cause reasoning performance to collapse on long-form reasoning tasks such as AIME. We propose \textit{LayerTrim}, a training-free structured pruning method that identifies redundant decoder layers and selectively trims their MLP features while preserving attention and decoder depth. LayerTrim is the first training-free structured pruning method designed to preserve long-form reasoning without gradients or recovery training. Across three reasoning language models and five benchmarks, LayerTrim outperforms existing structured pruning methods in most evaluated settings, improving average accuracy by up to 21.4 percentage points over RESP. On \texttt{Qwen3-14B}, LayerTrim achieves 1.24$\times$ faster prefill and 19.7\% lower peak GPU memory than the dense model, while better preserving reasoning ability. It also achieves the shortest average generation length among evaluated pruned models, while retaining the highest average accuracy. These results demonstrate that LayerTrim improves reasoning efficiency through both structural compression and reduced token generation.
Abstract:Structured pruning reduces LLM inference costs by removing model components to produce smaller dense models. However, existing methods can cause reasoning performance to collapse on long-form reasoning tasks such as AIME. We propose \textit{LayerTrim}, a training-free structured pruning method that identifies redundant decoder layers and selectively trims their MLP features while preserving attention and decoder depth. LayerTrim is the first training-free structured pruning method designed to preserve long-form reasoning without gradients or recovery training. Across three reasoning language models and five benchmarks, LayerTrim outperforms existing structured pruning methods in most evaluated settings, improving average accuracy by up to 21.4 percentage points over RESP. On \texttt{Qwen3-14B}, LayerTrim achieves 1.24$\times$ faster prefill and 19.7\% lower peak GPU memory than the dense model, while better preserving reasoning ability. It also achieves the shortest average generation length among evaluated pruned models, while retaining the highest average accuracy. These results demonstrate that LayerTrim improves reasoning efficiency through both structural compression and reduced token generation.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2610.19121 [cs.LG]
(or arXiv:2610.19121v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.19121 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 260 cuvinte
Subiecte conexe
[AINews] not much happened today
Latent Space · 10 sept. 2026
Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
arXiv · 29 iul. 2026
Built Technologies builds an AI-powered document intelligence solution on AWS to power agents across real estate finance
AWS · 15 iul. 2026
Looking Inside LLMs: Small-World Connectivity as a Signature of Reasoning Performance
arXiv · 10 oct. 2026