aiqbench analysis
This arXiv preprint uses computational mechanics to show that transformers trained on the joint behavior of interacting agents can represent each agent's beliefs in separate subspaces of the residual stream, with the representation growing linearly rather than exponentially in agent count. Causal interventions that insert counterfactuals reportedly steer individual agents' beliefs, which supports the claim that the factorization is functional rather than incidental. For interpretability researchers, the useful takeaway is the proposed framework for probing compositional structure in pretrained models, though the results so far come from controlled synthetic setups and need validation on realistic data.
Beliefs in feedback: Factored representations of interacting agents in transformers
arXiv:2611.02047v1 Announce Type: cross Abstract: Complex systems, from ecosystems to brains, are usually modeled as modular components coupled through feedback rather than as indivisible wholes. Can a transformer trained on the joint behavior of a group of interacting agents be understood in the same way? In this work, we develop a computational-mechanics framework to study agents interacting in feedback loops. We formally prove that any joint stochastic process can be presented as resulting from the behavior of agents interacting in feedback, and the resulting process is generated by a hidden Markov model whose Bayesian beliefs factorize into separate components per agent. Furthermore, our experiments show that transformers trained on such feedback sequences make use of this separability, disentangling the beliefs about the underlying factors into separate subspaces of their residual stream. This representation is found to scale linearly (instead of exponentially) with the number of agents. Moreover, interventions show that these representations are causally efficacious, allowing for targeted steering of individual agents' beliefs by selectively inserting counterfactuals. However, coarse-graining the behavior is found to disrupt the factorization, and can force a higher-dimensional representation. Together, the results establish both the dimensional advantage of a factorized representation and the ability of transformers to exploit it, laying the groundwork for investigating traces of compositional predictive architecture in pretrained models.
Abstract:Complex systems, from ecosystems to brains, are usually modeled as modular components coupled through feedback rather than as indivisible wholes. Can a transformer trained on the joint behavior of a group of interacting agents be understood in the same way? In this work, we develop a computational-mechanics framework to study agents interacting in feedback loops. We formally prove that any joint stochastic process can be presented as resulting from the behavior of agents interacting in feedback, and the resulting process is generated by a hidden Markov model whose Bayesian beliefs factorize into separate components per agent. Furthermore, our experiments show that transformers trained on such feedback sequences make use of this separability, disentangling the beliefs about the underlying factors into separate subspaces of their residual stream. This representation is found to scale linearly (instead of exponentially) with the number of agents. Moreover, interventions show that these representations are causally efficacious, allowing for targeted steering of individual agents' beliefs by selectively inserting counterfactuals. However, coarse-graining the behavior is found to disrupt the factorization, and can force a higher-dimensional representation. Together, the results establish both the dimensional advantage of a factorized representation and the ability of transformers to exploit it, laying the groundwork for investigating traces of compositional predictive architecture in pretrained models.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA); Adaptation and Self-Organizing Systems (nlin.AO)
arXiv:2611.02047 [cs.LG]
(or arXiv:2611.02047v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2611.02047 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 316 words
Related coverage
Understanding Rate-Distortion Performance in Distributed Transformer Inference
arXiv · Aug 4, 2026
Generalized belief states in transformers
LessWrong · Oct 31, 2026
Scaffold Then Internalize: Representation Injection for Diffusion Transformers
arXiv · Sep 30, 2026
Global Divergence, Local Convergence: Representation Geometry in SSMs and Transformers
arXiv · Sep 9, 2026