analiza aiqbench
This study reveals a fundamental temporal asymmetry in how decoder-only language models process factual recall, showing that relational constraints activate significantly earlier than specific entity commitments within the network depth. The findings demonstrate that while entity information is present early in the computation, its causal control over generation is deferred until later layers, challenging assumptions about uniform information flow during inference. This mechanistic insight into the routing of relation versus entity data offers critical value for researchers aiming to optimize model interpretability and improve factual accuracy through architectural adjustments.
Relation Before Entity: Deferred Commitment in Language Model Factual Recall
arXiv:2609.17537v1 Announce Type: new Abstract: We ask whether relation-type information (e.g., capital-of) and entity-specific information (e.g., France to Paris) become causally active at the final-token position at the same depth during recall. Using four complementary causal diagnostics across four decoder-only models and eight prompt families, we find a robust temporal asymmetry: relation information becomes generation-controlling before entity information does. Relation onset precedes entity onset by 10-16 tested layers (31-44% of network depth) at threshold 0.4, with the ordering holding across all 16 model-threshold combinations for thresholds 0.2-0.5. Critically, entity information is not absent early: entity-token patching succeeds at 90-100% in early layers. Instead, entity commitment to generation is deferred: entity information is available at the entity-token position but becomes generation-controlling at the final token only after being routed there.
Abstract:We ask whether relation-type information (e.g., capital-of) and entity-specific information (e.g., France to Paris) become causally active at the final-token position at the same depth during recall. Using four complementary causal diagnostics across four decoder-only models and eight prompt families, we find a robust temporal asymmetry: relation information becomes generation-controlling before entity information does. Relation onset precedes entity onset by 10-16 tested layers (31-44% of network depth) at threshold 0.4, with the ordering holding across all 16 model-threshold combinations for thresholds 0.2-0.5. Critically, entity information is not absent early: entity-token patching succeeds at 90-100% in early layers. Instead, entity commitment to generation is deferred: entity information is available at the entity-token position but becomes generation-controlling at the final token only after being routed there.
8 pages, 4 figures. Accepted at the Mechanistic Interpretability Workshop at the 43rd International Conference on Machine Learning (ICML 2026). Code available at this https URL
Computation and Language (cs.CL)
arXiv:2609.17537 [cs.CL]
(or arXiv:2609.17537v1 [cs.CL] for this version)
Text integral sub licența deschisă a publicației · 162 cuvinte
Subiecte conexe
The Continual Constitutional Convention: How language models revise the documents that govern them
AI Alignment Forum · 13 oct. 2026
Somewhat Contra Ted Chiang on AI Consciousness
LessWrong · 13 iun. 2026
A Red Line and Oversight Framework for Government AI Contracts
AI Alignment Forum · 18 iul. 2026
Deconfusing Corrigibility
AI Alignment Forum · 17 oct. 2026