2026 Summer PIBBSS Fellowship Symposium

When AI safety is discussed, people often use belief-desire terms: “the model wants to complete the task.” Yet no work has established whether transformers have a coherent belief-desire system. Applying the Jeffrey-Bolker framework to a task-trained toy model with ground-truth beliefs and desires, we find dissociable, causally used linear representations. There has been evidence that transformers represent belief states in their residual stream, when trained on synthetic data generated by HMMs. Yet the transformer architecture is not obviously suited to perform a Bayesian update algorithm. For some HMMs, previous work has shown how a single attention layer can give a good approximate computation of belief states. In this talk, I will describe another class of HMMs for which the attention mechanism…