aiqbench analysis
This research proposes Bank of Values (BoV), an architecture modification where deeper transformer layers utilize context-free value vectors stored as sparse parameters rather than deriving them from the residual stream. By decoupling these values, BoV reduces computational overhead and memory usage while matching or exceeding benchmark performance on models ranging from 135M to 780M parameters. The approach offers a significant efficiency gain for scaling large language models without sacrificing aggregate accuracy across diverse tasks.
Do Value Vectors in Deep Layers Need Context from the Residual Stream?
arXiv:2606.02780v3 Announce Type: replace Abstract: The success of the transformer architecture as the backbone of modern LLMs is in large part due to its use of attention layers. An attention layer follows the standard neural network paradigm: it takes the residual stream as input and thereby produces context-dependent query, key, and value vectors. However, we find that model performance meaningfully improves when deeper layers learn only a context-free value vector to preserve the original token information, without drawing on any context from the residual stream. When the model has access to this context-free value vector, adding back the context-dependent component provides little additional benefit for aggregate benchmark performance. Such context-free value vectors can be stored as sparse model parameters, eliminating the need to recompute or persistently cache these values. Through systematic ablations on the key design choices for such context-free value vectors, we propose Bank of Values (BoV), a new way of computing value vectors in attention by learning a lookup table of token-specific value vectors for each of the last third of layers. Across 135M and 780M models, BoV improves validation loss over standard attention and, at 780M, the average score across 21 benchmarks, matching the previous best method that adds token information to the value vector with less compute and memory.
Abstract:The success of the transformer architecture as the backbone of modern LLMs is in large part due to its use of attention layers. An attention layer follows the standard neural network paradigm: it takes the residual stream as input and thereby produces context-dependent query, key, and value vectors. However, we find that model performance meaningfully improves when deeper layers learn only a context-free value vector to preserve the original token information, without drawing on any context from the residual stream. When the model has access to this context-free value vector, adding back the context-dependent component provides little additional benefit for aggregate benchmark performance. Such context-free value vectors can be stored as sparse model parameters, eliminating the need to recompute or persistently cache these values. Through systematic ablations on the key design choices for such context-free value vectors, we propose Bank of Values (BoV), a new way of computing value vectors in attention by learning a lookup table of token-specific value vectors for each of the last third of layers. Across 135M and 780M models, BoV improves validation loss over standard attention and, at 780M, the average score across 21 benchmarks, matching the previous best method that adds token information to the value vector with less compute and memory.
13 pages, 5 figures. Code: this https URL
Computation and Language (cs.CL)
arXiv:2606.02780 [cs.CL]
(or arXiv:2606.02780v3 [cs.CL] for this version)
Full text under the publisher's open licence · 228 words
Related coverage
Do Value Vectors in Deep Layers Need Context from the Residual Stream?
arXiv · Jun 9, 2026
Do Value Vectors in Deep Layers Need Context from the Residual Stream?
arXiv · Jun 3, 2026
Some Intuitions on Steering Vectors
LessWrong · Sep 29, 2026
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · Oct 8, 2026