Learning What Matters: Supervising Sparse Attention Routing with Causal Evidence Sets

Abstract:Sparse attention reduces the cost of long contexts by allowing each query to read only selected parts of the input. These selectors are often trained by distilling the attention patterns of a dense teacher, assuming that attention reveals which context the teacher actually uses. We test that assumption on retrieval tasks where the evidence for each answer is known exactly. By masking parts of the context and measuring whether the answer changes, we find that attention and causal dependence often disagree, and distilled selectors inherit the mismatch. Teachers attend to outdated facts they have learned to ignore, and their attention can vary across training runs even when they rely on the same evidence. In a two-step reference task, attention at the answer skips the intermediate step because it was resolved earlier in the forward pass: a selector trained on attention achieves 41% accuracy, while the same selector trained on causal evidence reaches 99% and matches the teacher. These evidence sets require no annotation: recovered from a frozen teacher by masking alone, they train selectors to the same accuracy. We find the same conflict in pretrained models: Qwen2.5-3B gives more attention to an outdated fact than the current one on 58% of conflicting-fact examples despite answering correctly, while Gemma-2-9B rises from 56% to 99% accuracy when restricted to the two relevant sentences. Attention shows where a model looks, not necessarily what its answer depends on; across the regimes we tested, that dependence matched or outperformed attention as a training target.

Machine Learning (cs.LG); Computation and Language (cs.CL)

arXiv:2607.21692 [cs.LG]

(or arXiv:2607.21692v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2607.21692 Focus to learn more arXiv-issued DOI via DataCite

learning-what-matters/README.md

learning-what-matters/learning-what-matters/README.md

learning-what-matters/learning-what-matters/analysis/make_appendix.py

learning-what-matters/learning-what-matters/analysis/make_figures.py

learning-what-matters/learning-what-matters/analysis/make_main_tables.py

learning-what-matters/learning-what-matters/blocksmith_lm/__init__.py

learning-what-matters/learning-what-matters/blocksmith_lm/activation_taps.py

learning-what-matters/learning-what-matters/blocksmith_lm/adamw.py

learning-what-matters/learning-what-matters/blocksmith_lm/checkpoint.py

learning-what-matters/learning-what-matters/blocksmith_lm/cross_entropy_loss.py

learning-what-matters/learning-what-matters/blocksmith_lm/data_loading.py

learning-what-matters/learning-what-matters/blocksmith_lm/decode.py

learning-what-matters/learning-what-matters/blocksmith_lm/delta_rule_attention.py

learning-what-matters/learning-what-matters/blocksmith_lm/delta_rule_block.py

learning-what-matters/learning-what-matters/blocksmith_lm/delta_rule_lm.py

learning-what-matters/learning-what-matters/blocksmith_lm/dfa.py

learning-what-matters/learning-what-matters/blocksmith_lm/dfa_sampling.py

learning-what-matters/learning-what-matters/blocksmith_lm/embedding.py

learning-what-matters/learning-what-matters/blocksmith_lm/gradient_clipping.py

learning-what-matters/learning-what-matters/blocksmith_lm/learning_rate_scheduler.py

learning-what-matters/learning-what-matters/blocksmith_lm/linear.py

learning-what-matters/learning-what-matters/blocksmith_lm/llm.py

learning-what-matters/learning-what-matters/blocksmith_lm/looped_transformer_lm.py

learning-what-matters/learning-what-matters/blocksmith_lm/lstm_lm.py

learning-what-matters/learning-what-matters/blocksmith_lm/mlp_lm.py

learning-what-matters/learning-what-matters/blocksmith_lm/multihead_self_attention.py

learning-what-matters/learning-what-matters/blocksmith_lm/multihead_self_attention_with_rope.py

learning-what-matters/learning-what-matters/blocksmith_lm/rmsnorm.py

learning-what-matters/learning-what-matters/blocksmith_lm/rotary_positional_embedding.py

learning-what-matters/learning-what-matters/blocksmith_lm/scaled_dot_product_attention.py

learning-what-matters/learning-what-matters/blocksmith_lm/sgd.py

learning-what-matters/learning-what-matters/blocksmith_lm/softmax.py

learning-what-matters/learning-what-matters/blocksmith_lm/swiglu.py

learning-what-matters/learning-what-matters/blocksmith_lm/tokenizer.py

learning-what-matters/learning-what-matters/blocksmith_lm/train_bpe.py

learning-what-matters/learning-what-matters/blocksmith_lm/training_plots.py

learning-what-matters/learning-what-matters/blocksmith_lm/transformer_block.py

learning-what-matters/learning-what-matters/blocksmith_lm/transformer_lm.py

learning-what-matters/learning-what-matters/learning_what_matters/__init__.py

learning-what-matters/learning-what-matters/learning_what_matters/ablation.py

learning-what-matters/learning-what-matters/learning_what_matters/agreement.py

learning-what-matters/learning-what-matters/learning_what_matters/attention.py

learning-what-matters/learning-what-matters/learning_what_matters/harness.py

learning-what-matters/learning-what-matters/learning_what_matters/labels.py

learning-what-matters/learning-what-matters/learning_what_matters/pretrained.py

learning-what-matters/learning-what-matters/learning_what_matters/routers.py

learning-what-matters/learning-what-matters/learning_what_matters/tasks.py

learning-what-matters/learning-what-matters/learning_what_matters/tasks_hard.py

learning-what-matters/learning-what-matters/learning_what_matters/vocab.py

learning-what-matters/learning-what-matters/pyproject.toml

learning-what-matters/learning-what-matters/rebuild_all.sh

learning-what-matters/learning-what-matters/reproduce.sh

learning-what-matters/learning-what-matters/runs/agreement/base/report.json

learning-what-matters/learning-what-matters/runs/agreement/base5/report.json

learning-what-matters/learning-what-matters/runs/agreement/extended/report.json

learning-what-matters/learning-what-matters/runs/agreement/greedy_chains.json

learning-what-matters/learning-what-matters/runs/diagnostics/chain_hard_causal.json

learning-what-matters/learning-what-matters/runs/diagnostics/chain_hard_imitation.json

learning-what-matters/learning-what-matters/runs/diagnostics/chain_router_grid.json

learning-what-matters/learning-what-matters/runs/diagnostics/chain_router_imitation.json

learning-what-matters/learning-what-matters/runs/diagnostics/chain_router_transfer.json

learning-what-matters/learning-what-matters/runs/diagnostics/hardened_dense_baselines.json

learning-what-matters/learning-what-matters/runs/diagnostics/multihop_router.json

learning-what-matters/learning-what-matters/runs/diagnostics/oracle_routing.json

learning-what-matters/learning-what-matters/runs/diagnostics/recovered_synth.json

learning-what-matters/learning-what-matters/runs/diagnostics/recovered_synth_budget.json

learning-what-matters/learning-what-matters/runs/diagnostics/recovered_synth_delta.json

learning-what-matters/learning-what-matters/runs/diagnostics/recovered_synth_floor.json

learning-what-matters/learning-what-matters/runs/diagnostics/recovered_synth_gradient.json

learning-what-matters/learning-what-matters/runs/diagnostics/recovered_synth_hybrid.json

learning-what-matters/learning-what-matters/runs/diagnostics/recovered_synth_verified.json

learning-what-matters/learning-what-matters/runs/diagnostics/recovery_rules.json

learning-what-matters/learning-what-matters/runs/diagnostics/router_bench.json

learning-what-matters/learning-what-matters/runs/pretrained/Qwen2.5-0.5B-Instruct.json

learning-what-matters/learning-what-matters/runs/pretrained/Qwen2.5-1.5B-Instruct.json

learning-what-matters/learning-what-matters/runs/pretrained/Qwen2.5-14B-Instruct.json

learning-what-matters/learning-what-matters/runs/pretrained/Qwen2.5-3B-Instruct.json

learning-what-matters/learning-what-matters/runs/pretrained/Qwen2.5-7B-Instruct.json

learning-what-matters/learning-what-matters/runs/pretrained/Yi-1.5-9B-Chat.json

learning-what-matters/learning-what-matters/runs/pretrained/gemma-2-9b-it.json

learning-what-matters/learning-what-matters/runs/pretrained/overwrite_Qwen2.5-3B-Instruct.json

learning-what-matters/learning-what-matters/runs/pretrained/overwrite_Yi-1.5-9B-Chat.json

learning-what-matters/learning-what-matters/runs/pretrained/overwrite_gemma-2-9b-it.json

learning-what-matters/learning-what-matters/runs/pretrained/real_qa.json

learning-what-matters/learning-what-matters/runs/pretrained/real_qa_typematched.json

learning-what-matters/learning-what-matters/runs/pretrained/routing_Qwen2.5-3B-Instruct.json

learning-what-matters/learning-what-matters/runs/pretrained/routing_Qwen2.5-7B-Instruct.json

learning-what-matters/learning-what-matters/runs/pretrained/routing_Yi-1.5-9B-Chat.json

learning-what-matters/learning-what-matters/runs/pretrained/routing_gemma-2-9b-it.json

learning-what-matters/learning-what-matters/runs/pretrained/routing_recovered_Qwen2.5-3B-Instruct.json

learning-what-matters/learning-what-matters/runs/pretrained/scaling_recovered_128.json

learning-what-matters/learning-what-matters/runs/pretrained/scaling_recovered_32.json

learning-what-matters/learning-what-matters/runs/pretrained/scaling_recovered_64.json

learning-what-matters/learning-what-matters/runs/routers/ablation/results.json

learning-what-matters/learning-what-matters/runs/routers/base/results.json

learning-what-matters/learning-what-matters/runs/routers/base5/results.json

learning-what-matters/learning-what-matters/runs/routers/base_3x/results.json

learning-what-matters/learning-what-matters/runs/routers/base_lr3e3/results.json

learning-what-matters/learning-what-matters/runs/routers/base_lr3e4/results.json

learning-what-matters/learning-what-matters/runs/routers/extended/results.json

learning-what-matters/learning-what-matters/runs/teachers/base/seed0/anchors.json

learning-what-matters/learning-what-matters/runs/teachers/base/seed0/config.json

learning-what-matters/learning-what-matters/runs/teachers/base/seed0/metrics.json

learning-what-matters/learning-what-matters/runs/teachers/base/seed1/config.json

learning-what-matters/learning-what-matters/runs/teachers/base/seed1/metrics.json

learning-what-matters/learning-what-matters/runs/teachers/base/seed2/config.json

learning-what-matters/learning-what-matters/runs/teachers/base/seed2/metrics.json

learning-what-matters/learning-what-matters/runs/teachers/base/seed3/config.json

learning-what-matters/learning-what-matters/runs/teachers/base/seed3/metrics.json

learning-what-matters/learning-what-matters/runs/teachers/base/seed4/config.json

learning-what-matters/learning-what-matters/runs/teachers/base/seed4/metrics.json

learning-what-matters/learning-what-matters/runs/teachers/extended/seed0/budget_curve.json

learning-what-matters/learning-what-matters/runs/teachers/extended/seed0/config.json

learning-what-matters/learning-what-matters/runs/teachers/extended/seed0/metrics.json

learning-what-matters/learning-what-matters/runs/teachers/extended/seed1/budget_curve.json

learning-what-matters/learning-what-matters/runs/teachers/extended/seed1/config.json

learning-what-matters/learning-what-matters/runs/teachers/extended/seed1/metrics.json

learning-what-matters/learning-what-matters/runs/teachers/extended/seed2/config.json

learning-what-matters/learning-what-matters/runs/teachers/extended/seed2/metrics.json

learning-what-matters/learning-what-matters/runs/teachers/extended/seed3/config.json

learning-what-matters/learning-what-matters/runs/teachers/extended/seed3/metrics.json

learning-what-matters/learning-what-matters/runs/teachers/extended/seed4/config.json

learning-what-matters/learning-what-matters/runs/teachers/extended/seed4/metrics.json

learning-what-matters/learning-what-matters/runs/teachers/hardened/seed0/config.json

learning-what-matters/learning-what-matters/runs/teachers/hardened/seed0/metrics.json

learning-what-matters/learning-what-matters/runs/teachers/hardened/seed1/config.json

learning-what-matters/learning-what-matters/runs/teachers/hardened/seed1/metrics.json

learning-what-matters/learning-what-matters/runs/teachers/lp_lr1e3/seed0/config.json

learning-what-matters/learning-what-matters/runs/teachers/lp_lr1e3/seed0/metrics.json

learning-what-matters/learning-what-matters/runs/teachers/lp_lr1e4/seed0/config.json

learning-what-matters/learning-what-matters/runs/teachers/lp_lr1e4/seed0/metrics.json

learning-what-matters/learning-what-matters/runs/teachers/smoke/seed0/config.json

learning-what-matters/learning-what-matters/runs/teachers/smoke/seed0/metrics.json

learning-what-matters/learning-what-matters/scripts/agreement_report.py

learning-what-matters/learning-what-matters/scripts/analyze_recovery.py

learning-what-matters/learning-what-matters/scripts/anchors.py

learning-what-matters/learning-what-matters/scripts/bench_router.py

learning-what-matters/learning-what-matters/scripts/budget_curve.py

learning-what-matters/learning-what-matters/scripts/diag_aggregation.py

learning-what-matters/learning-what-matters/scripts/diag_ar.py

learning-what-matters/learning-what-matters/scripts/diag_multihop.py

learning-what-matters/learning-what-matters/scripts/greedy_chains.py

learning-what-matters/learning-what-matters/scripts/hardened_dense.py

learning-what-matters/learning-what-matters/scripts/oracle_routing.py

learning-what-matters/learning-what-matters/scripts/overwrite_intervention.py

learning-what-matters/learning-what-matters/scripts/pretrained_probe.py

learning-what-matters/learning-what-matters/scripts/pretrained_recovered.py

learning-what-matters/learning-what-matters/scripts/pretrained_router.py

learning-what-matters/learning-what-matters/scripts/real_qa.py

learning-what-matters/learning-what-matters/scripts/train_chain_router.py

learning-what-matters/learning-what-matters/scripts/train_recovered_synth.py

learning-what-matters/learning-what-matters/scripts/train_routers.py

learning-what-matters/learning-what-matters/scripts/train_teachers.py

learning-what-matters/learning-what-matters/tests/test_ablation.py

learning-what-matters/learning-what-matters/tests/test_attention.py

learning-what-matters/learning-what-matters/tests/test_labels.py

learning-what-matters/learning-what-matters/tests/test_routers.py

learning-what-matters/learning-what-matters/tests/test_tasks.py

learning-what-matters/learning-what-matters/tests/test_tasks_hard.py

learning-what-matters/requirements.txt

(158 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.