analiza aiqbench
Locality-Aware Redundancy Pruning (LoRP) introduces a training-free method to optimize large language models by identifying and removing redundant layers based on representation locality rather than fixed assumptions. By clustering hidden states, this approach improves inference efficiency while maintaining or enhancing accuracy across diverse model architectures. The technique addresses a core challenge in deploying LLMs: reducing computational cost without sacrificing performance through architectural redundancy.
Locality-Aware Redundancy Pruning for LLM Depth Compression
arXiv:2605.27786v3 Announce Type: replace-cross Abstract: Large language models are known to contain representational redundancy across network depth, making depth pruning an effective approach for improving inference efficiency. Existing one-shot pruning methods rely on local layer importance or fixed redundancy assumptions across architectures. We propose Locality-Aware Redundancy Pruning (LoRP), a training-free one-shot depth pruning framework guided by representation locality. We show that inter-layer redundancy can be either localized or globally distributed depending on the LLM architecture. To characterize this phenomenon, we introduce Representation Locality Score (RLS), derived from global inter-layer hidden-state similarity. Using a small calibration set, LoRP computes pairwise layer similarity, clusters layers by representational similarity, and allocates pruning according to residual intra-cluster redundancy. Experiments across diverse LLM families show improvements in both perplexity and downstream task accuracy. Official github repository: https://github.com/daniel-eai/LoRP-Locality-Aware-Redundancy-Pruning/
Abstract:Large language models are known to contain representational redundancy across network depth, making depth pruning an effective approach for improving inference efficiency. Existing one-shot pruning methods rely on local layer importance or fixed redundancy assumptions across architectures. We propose Locality-Aware Redundancy Pruning (LoRP), a training-free one-shot depth pruning framework guided by representation locality. We show that inter-layer redundancy can be either localized or globally distributed depending on the LLM architecture. To characterize this phenomenon, we introduce Representation Locality Score (RLS), derived from global inter-layer hidden-state similarity. Using a small calibration set, LoRP computes pairwise layer similarity, clusters layers by representational similarity, and allocates pruning according to residual intra-cluster redundancy. Experiments across diverse LLM families show improvements in both perplexity and downstream task accuracy. Official github repository: this https URL
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2605.27786 [cs.LG]
(or arXiv:2605.27786v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.27786 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 224 cuvinte
Subiecte conexe
Locality-Aware Redundancy Pruning for LLM Depth Compression
arXiv · 9 iun. 2026
RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT
arXiv · 9 iun. 2026
Beyond Layer Importance in Layer-wise Sparsity: An Inter-Layer Perturbation-Absorption Perspective
arXiv · 17 iun. 2026
Mapping Text to Multiplex Graph: Prompt Compression as L\'evy Walk-Guided Graph Pruning
arXiv · 16 sept. 2026