aiqbench analysis
This research investigates the limitations of large language models in accurately predicting how their own outputs change when prompts are modified, revealing systematic errors in counterfactual reasoning. The authors propose a scalable synthetic-data pipeline combined with reinforcement learning to enhance these self-modeling capabilities across multiple open-source model families. However, the study cautions that these performance gains do not necessarily indicate true introspection or privileged access to internal decision processes.
Evaluating and Improving LLM Self-Modeling
arXiv:2608.30980v1 Announce Type: cross Abstract: We study self-modeling: an LLM's ability to answer questions about its own behavior. We focus on verifiable behavioral questions, such as whether a prompt edit would change the model's final answer. To measure this capability, we introduce a benchmark that tests diverse types of self-modeling questions. Current models show non-trivial but limited self-modeling skill, and make systematic mistakes on simple counterfactual questions about their own behavior. To improve self-modeling skill, we develop a scalable synthetic-data pipeline that produces self-modeling training data, and show that reinforcement-learning can improve aggregate self-modeling skill across three open-source model families with some transfer to held-out tasks. These gains, however, do not seem to constitute introspection consistently: improved self-modeling may not arise from privileged access to the model's internal decision process.
Abstract:We study self-modeling: an LLM's ability to answer questions about its own behavior. We focus on verifiable behavioral questions, such as whether a prompt edit would change the model's final answer. To measure this capability, we introduce a benchmark that tests diverse types of self-modeling questions. Current models show non-trivial but limited self-modeling skill, and make systematic mistakes on simple counterfactual questions about their own behavior. To improve self-modeling skill, we develop a scalable synthetic-data pipeline that produces self-modeling training data, and show that reinforcement-learning can improve aggregate self-modeling skill across three open-source model families with some transfer to held-out tasks. These gains, however, do not seem to constitute introspection consistently: improved self-modeling may not arise from privileged access to the model's internal decision process.
89 pages, 25 figures. Published as a conference paper at EMNLP '26 (Main)
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2608.30980 [cs.CL]
(or arXiv:2608.30980v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.30980 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 236 words
Related coverage
[AINews] Reality Checks on AI News (Yegge shuts down Gas Town, Databricks’ +60% Astra cost)
Latent Space · Sep 17, 2026
Projection sampling preserves capabilities but diminishes gains from alignment fine-tuning
LessWrong · Oct 16, 2026
[AINews] not much happened today
Latent Space · Sep 18, 2026
Self-Modeling Interventions Modulate Emergent Misalignment
LessWrong · Oct 5, 2026