aiqbench analysis
This paper introduces K-Token Merging, a novel framework that compresses input sequences by merging contiguous blocks of token embeddings within the latent space rather than at the token level. By utilizing a lightweight encoder and LoRA adaptation, the method achieves up to 75% reduction in input length while maintaining performance on tasks ranging from structural reasoning to code editing. This approach directly addresses the quadratic scaling bottleneck of self-attention mechanisms in large language models, offering a significant pathway for handling long-context scenarios efficiently.
Compressing Sequences in the Latent Embedding Space: $K$-Token Merging for Large Language Models
arXiv:2604.15153v3 Announce Type: replace Abstract: Large Language Models (LLMs) incur significant computational and memory costs when processing long prompts, as full self-attention scales quadratically with input length. Token compression aims to address this challenge by reducing the number of tokens representing inputs. However, existing prompt-compression approaches primarily operate in token space and overlook inefficiencies in the latent embedding space. In this paper, we propose K-Token Merging, a latent-space compression framework that merges each contiguous block of K token embeddings into a single embedding via a lightweight encoder. The compressed sequence is processed by a LoRA-adapted LLM, while generation remains in the original vocabulary. Experiments on structural reasoning (Textualized Tree), sentiment classification (Amazon Reviews), and code editing (CommitPackFT) show that K-Token Merging lies on the Pareto frontier of performance vs. compression, achieving up to 75% input length reduction with minimal performance degradation. Code is available at https://github.com/shsjxzh/K-Token-Merging.
Abstract:Large Language Models (LLMs) incur significant computational and memory costs when processing long prompts, as full self-attention scales quadratically with input length. Token compression aims to address this challenge by reducing the number of tokens representing inputs. However, existing prompt-compression approaches primarily operate in token space and overlook inefficiencies in the latent embedding space. In this paper, we propose K-Token Merging, a latent-space compression framework that merges each contiguous block of K token embeddings into a single embedding via a lightweight encoder. The compressed sequence is processed by a LoRA-adapted LLM, while generation remains in the original vocabulary. Experiments on structural reasoning (Textualized Tree), sentiment classification (Amazon Reviews), and code editing (CommitPackFT) show that K-Token Merging lies on the Pareto frontier of performance vs. compression, achieving up to 75% input length reduction with minimal performance degradation. Code is available at this https URL.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2604.15153 [cs.CL]
(or arXiv:2604.15153v3 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2604.15153 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 239 words
Related coverage
Rethinking Sequence Compression with Test-time Regression
arXiv · Oct 23, 2026
Denoising Blocks, Not Tokens: Efficient Compressed Continuous Diffusion with Branching Token Realization
arXiv · Oct 8, 2026
SequenceO1: End-to-End Ultra-Long (100K) Sequence Modeling in Recommendation with Low-Rank Caching
arXiv · Sep 9, 2026
MWOP: Modality-aware Width-wise Operation Pruning for Efficient MLLMs
arXiv · Oct 2, 2026