aiqbench analysis
This research investigates how small-scale transformers can internally simulate Bayesian reasoning to infer latent common causes across different contexts. By disentangling model mechanisms from data structure, the study offers a tractable framework for understanding cross-contextual generalization in natural language prediction tasks. This work advances fundamental machine learning theory by demonstrating that complex probabilistic inference can emerge within constrained architectures.
Small transformers track Bayesian evidence for latent common causes via a context-invariant mechanism
arXiv:2609.35161v1 Announce Type: new Abstract: We present an in-depth investigation of how a form of Bayesian reasoning about common causes can emerge as a cross-contextual generalization in small, tractable transformers. Incrementing on recent work, our set-up (i) disentangles causal mechanisms in the model from the causal structure of the true data-generating process, (ii) orients more towards natural language prediction by considering inference of latent common causes, and (iii) considers whether and how Bayesian evidence accumulation for latent common causes can be implemented in representations and mechanisms that allow for cross-context generalization to novel test cases.
Abstract:We present an in-depth investigation of how a form of Bayesian reasoning about common causes can emerge as a cross-contextual generalization in small, tractable transformers. Incrementing on recent work, our set-up (i) disentangles causal mechanisms in the model from the causal structure of the true data-generating process, (ii) orients more towards natural language prediction by considering inference of latent common causes, and (iii) considers whether and how Bayesian evidence accumulation for latent common causes can be implemented in representations and mechanisms that allow for cross-context generalization to novel test cases.
Machine Learning (cs.LG)
arXiv:2609.35161 [cs.LG]
(or arXiv:2609.35161v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.35161 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 184 words
