aiqbench analysis
This arXiv paper proposes TTR-Compressor, which compresses token sequences by fitting a regression problem whose closed-form solution serves as the compressed representation, with the regression trained end-to-end. The authors report gains over attention-based compressors on RULER for block-wise sparse attention, and for multimodal models they match an MLP visual-token merger while using 55.6% fewer visual tokens. These are single-paper results on specific benchmarks, so readers should watch for independent replication and for how the method's compute cost compares in practice.
Rethinking Sequence Compression with Test-time Regression
arXiv:2610.31574v1 Announce Type: new Abstract: Sequence compression is a fundamental operation and a critical information bottleneck in efficient language and multimodal models, and yet prevailing approaches based on pooling or learned projections remain underperforming. We revisit sequence compression through the lens of test-time regression: input tokens define a regression problem whose fitted coefficients form the compressed representation. Importantly, the regression problem is learned end-to-end via backpropagation, while its coefficients are computed efficiently in closed form during the forward pass. This leads to TTR-Compressor, a general alternative to conventional pooling and projection-based compression. We evaluate TTR-Compressor as the compressor for block-wise sparse attention and the visual token merger for multimodal large language models. On block-wise sparse attention, TTR-Compressor improves RULER performance over attention-based compressors by a large margin. For visual token merging, it consistently outperforms the standard MLP merger and matches its performance with 55.6% fewer visual tokens. These results establish test-time regression as an effective primitive for sequence compression in efficient language and multimodal models.
Abstract:Sequence compression is a fundamental operation and a critical information bottleneck in efficient language and multimodal models, and yet prevailing approaches based on pooling or learned projections remain underperforming. We revisit sequence compression through the lens of test-time regression: input tokens define a regression problem whose fitted coefficients form the compressed representation. Importantly, the regression problem is learned end-to-end via backpropagation, while its coefficients are computed efficiently in closed form during the forward pass. This leads to TTR-Compressor, a general alternative to conventional pooling and projection-based compression. We evaluate TTR-Compressor as the compressor for block-wise sparse attention and the visual token merger for multimodal large language models. On block-wise sparse attention, TTR-Compressor improves RULER performance over attention-based compressors by a large margin. For visual token merging, it consistently outperforms the standard MLP merger and matches its performance with 55.6% fewer visual tokens. These results establish test-time regression as an effective primitive for sequence compression in efficient language and multimodal models.
Machine Learning (cs.LG)
arXiv:2610.31574 [cs.LG]
(or arXiv:2610.31574v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.31574 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 254 words
Related coverage
Rethinking Video Token Compression with a Global Codebook: Learning Once, Compressing Everywhere
arXiv · Aug 5, 2026
Rethinking LoRA Memory Through the Lens of KV Cache Compression
arXiv · Jun 5, 2026
From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression
arXiv · Aug 27, 2026
Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression
arXiv · Jun 3, 2026