aiqbench analysis
This research demonstrates the ability to forecast Transformer model behavior on out-of-distribution data by analyzing attention patterns observed during standard training. The study decouples mechanistic faithfulness from predictive utility, revealing that internal representations can signal generalization rules even when causal links are absent. This proof-of-concept establishes a new interpretability objective focused on assessing reliability under distribution shift rather than just explaining in-distribution decisions.
Can Interpretation Predict Behavior on Unseen Data?
arXiv:2507.06445v2 Announce Type: replace-cross Abstract: Interpretability research often predicts model responses to targeted mechanistic interventions. But can we predict responses to unseen input data? We propose and demonstrate this alternate objective by using model internals to predict their out-of-distribution (OOD) behavior. We train hundreds of Transformers on simple synthetic tasks, where perfect in-distribution accuracy is compatible with multiple OOD generalization rules. We successfully use attention patterns -- observed only on in-distribution data -- to predict which rule each model follows on OOD data. Our experiments decouple the mechanistic faithfulness of our interpretation from its predictive value; ablations reveal such internal patterns can suppress rather than support the rule they predict, showing observational analysis can forecast behavior even when causal analysis fails to support a simple cause-effect link. Our findings are a proof-of-concept for a new interpretability objective: understanding model internals to predict behavior and assess reliability under distribution shift.
Title:Can Interpretation Predict Behavior on Unseen Data?
Abstract:Interpretability research often predicts model responses to targeted mechanistic interventions. But can we predict responses to unseen input data? We propose and demonstrate this alternate objective by using model internals to predict their out-of-distribution (OOD) behavior. We train hundreds of Transformers on simple synthetic tasks, where perfect in-distribution accuracy is compatible with multiple OOD generalization rules. We successfully use attention patterns -- observed only on in-distribution data -- to predict which rule each model follows on OOD data. Our experiments decouple the mechanistic faithfulness of our interpretation from its predictive value; ablations reveal such internal patterns can suppress rather than support the rule they predict, showing observational analysis can forecast behavior even when causal analysis fails to support a simple cause-effect link. Our findings are a proof-of-concept for a new interpretability objective: understanding model internals to predict behavior and assess reliability under distribution shift.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2507.06445 [cs.LG]
(or arXiv:2507.06445v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2507.06445 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 250 words
Related coverage
Building and evaluating model diffing agents
AI Alignment Forum · Jun 12, 2026
A Rolling-Window Framework for Churn Prediction and Behavioral Driver Identification
arXiv · Jun 8, 2026
Sim2Win: A Team-Agnostic, Event-Based Pre-Match Outcome Prediction and Tactical Profiling System for Football
arXiv · Jul 31, 2026
Learning to Simulate Individuals from Macro Social Signals
arXiv · Oct 7, 2026