aiqbench analysis
TechPulse take: This research proposes Bank of Values (BoV), an architecture modification where deep transformer layers utilize context-free value vectors stored as sparse parameters to preserve token information. The approach demonstrates that removing context-dependent components from these specific layers yields measurable improvements in validation loss and benchmark scores while reducing compute and memory overhead. For engineers optimizing large language models, this offers a viable path to enhancing efficiency without sacrificing aggregate performance.
Do Value Vectors in Deep Layers Need Context from the Residual Stream?
arXiv:2606.02780v1 Announce Type: new Abstract: The success of the transformer architecture as the backbone of modern LLMs is in large part due to its use of attention layers. An attention layer follows the standard neural network paradigm: it takes the residual stream as input and thereby produces context-dependent query, key, and value vectors. However, we find that model performance meaningfully improves when deeper layers learn only a context-free value vector to preserve the original token information, without drawing on any context from the residual stream. When the model has access to this context-free value vector, adding back the context-dependent component provides little additional benefit for aggregate benchmark performance. Such context-free value vectors can be stored as sparse model parameters, eliminating the need to recompute or persistently cache these values. Through systematic ablations on the key design choices for such context-free value vectors, we propose Bank of Values (BoV), a new way of computing value vectors in attention by learning a lookup table of token-specific value vectors for each of the last third of layers. Across 135M and 780M models, BoV improves validation loss over standard attention and, at 780M, the average score across 21 benchmarks, matching the previous best method that adds token information to the value vector with less compute and memory.
Title:Do Value Vectors in Deep Layers Need Context from the Residual Stream?
Abstract:The success of the transformer architecture as the backbone of modern LLMs is in large part due to its use of attention layers. An attention layer follows the standard neural network paradigm: it takes the residual stream as input and thereby produces context-dependent query, key, and value vectors. However, we find that model performance meaningfully improves when deeper layers learn only a context-free value vector to preserve the original token information, without drawing on any context from the residual stream. When the model has access to this context-free value vector, adding back the context-dependent component provides little additional benefit for aggregate benchmark performance. Such context-free value vectors can be stored as sparse model parameters, eliminating the need to recompute or persistently cache these values. Through systematic ablations on the key design choices for such context-free value vectors, we propose Bank of Values (BoV), a new way of computing value vectors in attention by learning a lookup table of token-specific value vectors for each of the last third of layers. Across 135M and 780M models, BoV improves validation loss over standard attention and, at 780M, the average score across 21 benchmarks, matching the previous best method that adds token information to the value vector with less compute and memory.
13 pages, 5 figures. Code: this https URL
Computation and Language (cs.CL)
arXiv:2606.02780 [cs.CL]
(or arXiv:2606.02780v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.02780 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 323 words
Related coverage
Do Value Vectors in Deep Layers Need Context from the Residual Stream?
arXiv · Jun 9, 2026
Some Intuitions on Steering Vectors
LessWrong · Sep 29, 2026
Do Value Vectors in Deep Layers Need Context from the Residual Stream?
arXiv · Jul 21, 2026
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · Oct 8, 2026