aiqbench analysis
This research addresses the memory bottleneck in long-context LLM inference by introducing RoPE-Aligned Pruning (RAP), which preserves positional semantics during KV-cache compression. By constraining pruning to Rotary Position Embedding-aligned pairs, RAP achieves significant dimensionality reduction while maintaining model accuracy across various architectures like Llama and Mistral. The method offers a practical pathway for optimizing memory efficiency without sacrificing the performance gains of modern position embeddings.
RAP: KV-Cache Compression via RoPE-Aligned Pruning
arXiv:2602.02599v4 Announce Type: replace-cross Abstract: Long-context inference in large language models (LLMs) is bottlenecked by the memory and compute of the key-value (KV) cache. Structured pruning is a direct way to shrink it: dropping the least useful channels of the W_k, W_v projection weights to reduce the output KV dimensions. However, modern LLMs apply Rotary Position Embedding (RoPE) after the QK projections, which rotates feature dimensions in pairs. Therefore, removing individual channels breaks these pairs, corrupting RoPE's positional semantics and rendering the pruned model unusable. We propose RoPE-Aligned Pruning (RAP), which constrains the pruning granularity to RoPE-aligned pairs rather than individual channels: removing whole pairs to keep the rotation intact. Our evaluation across Llama, Mistral, and Qwen models from 3B to 14B shows that RAP preserves accuracy at 30% KV compression (retain ratio \r{ho} = 0.7), far outperforms RoPE-blind channel pruning, stays near the strongest low-rank method at lower attention cost, and composes with orthogonal methods such as quantization.
Abstract:Long-context inference in large language models (LLMs) is bottlenecked by the memory and compute of the key-value (KV) cache. Structured pruning is a direct way to shrink it: dropping the least useful channels of the W_k, W_v projection weights to reduce the output KV dimensions. However, modern LLMs apply Rotary Position Embedding (RoPE) after the QK projections, which rotates feature dimensions in pairs. Therefore, removing individual channels breaks these pairs, corrupting RoPE's positional semantics and rendering the pruned model unusable. We propose RoPE-Aligned Pruning (RAP), which constrains the pruning granularity to RoPE-aligned pairs rather than individual channels: removing whole pairs to keep the rotation intact. Our evaluation across Llama, Mistral, and Qwen models from 3B to 14B shows that RAP preserves accuracy at 30% KV compression (retain ratio \r{ho} = 0.7), far outperforms RoPE-blind channel pruning, stays near the strongest low-rank method at lower attention cost, and composes with orthogonal methods such as quantization.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2602.02599 [cs.LG]
(or arXiv:2602.02599v4 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2602.02599 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 249 words
Related coverage
Dual-QK: Sharp Queries and Flat Keys for Prunable 2-bit KV Caches
arXiv · Oct 8, 2026
QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching
arXiv · Oct 1, 2026
CHASE: Channel-Aligned Structure Exploitation for Geometry-Aware Model Engineering
arXiv · Oct 8, 2026
A Self-Pruning Transformer: Extreme KV-Cache Compression with Universal Attention
arXiv · Oct 8, 2026