analiza aiqbench
This research proposes Bank of Values (BoV), an architecture modification where deeper transformer layers utilize context-free value vectors stored as sparse parameters rather than deriving them from the residual stream. By decoupling these values, BoV reduces computational overhead and memory usage while matching or exceeding benchmark performance on models ranging from 135M to 780M parameters. The approach offers a significant efficiency gain for scaling large language models without sacrificing aggregate accuracy across diverse tasks.
Do Value Vectors in Deep Layers Need Context from the Residual Stream?
arXiv:2606.02780v3 Announce Type: replace Abstract: The success of the transformer architecture as the backbone of modern LLMs is in large part due to its use of attention layers. An attention layer follows the standard neural network paradigm: it takes the residual stream as input and thereby produces context-dependent query, key, and value vectors. However, we find that model performance meaningfully improves when deeper layers learn only a context-free value vector to preserve the original token information, without drawing on any context from the residual stream. When the model has access to this context-free value vector, adding back the context-dependent component provides little additional benefit for aggregate benchmark performance. Such context-free value vectors can be stored as sparse model parameters, eliminating the need to recompute or persistently cache these values. Through systematic ablations on the key design choices for such context-free value vectors, we propose Bank of Values (BoV), a new way of computing value vectors in attention by learning a lookup table of token-specific value vectors for each of the last third of layers. Across 135M and 780M models, BoV improves validation loss over standard attention and, at 780M, the average score across 21 benchmarks, matching the previous best method that adds token information to the value vector with less compute and memory.
Abstract:The success of the transformer architecture as the backbone of modern LLMs is in large part due to its use of attention layers. An attention layer follows the standard neural network paradigm: it takes the residual stream as input and thereby produces context-dependent query, key, and value vectors. However, we find that model performance meaningfully improves when deeper layers learn only a context-free value vector to preserve the original token information, without drawing on any context from the residual stream. When the model has access to this context-free value vector, adding back the context-dependent component provides little additional benefit for aggregate benchmark performance. Such context-free value vectors can be stored as sparse model parameters, eliminating the need to recompute or persistently cache these values. Through systematic ablations on the key design choices for such context-free value vectors, we propose Bank of Values (BoV), a new way of computing value vectors in attention by learning a lookup table of token-specific value vectors for each of the last third of layers. Across 135M and 780M models, BoV improves validation loss over standard attention and, at 780M, the average score across 21 benchmarks, matching the previous best method that adds token information to the value vector with less compute and memory.
13 pages, 5 figures. Code: this https URL
Computation and Language (cs.CL)
arXiv:2606.02780 [cs.CL]
(or arXiv:2606.02780v3 [cs.CL] for this version)
Text integral sub licența deschisă a publicației · 228 cuvinte
Subiecte conexe
Do Value Vectors in Deep Layers Need Context from the Residual Stream?
arXiv · 9 iun. 2026
Do Value Vectors in Deep Layers Need Context from the Residual Stream?
arXiv · 3 iun. 2026
Some Intuitions on Steering Vectors
LessWrong · 29 sept. 2026
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · 8 oct. 2026