The separation principle: where do beliefs and desires come from?

TLDR: Psychology, economics, and other disciplines describe agents as systems driven by beliefs and desires. This post argues that the belief-desire view can be derived from classic theorems from optimal control and reinforcement learning. This suggests seeing beliefs and desires as properties of optimal policies rather than as assumptions from folk psychology. Introduction One way to think about agents is as "systems that act for reasons".[1] This compact statement can be interpreted as encapsulating two key implications: The notion of action assumes a boundary between agent and environment, so that the former can act on the latter. The term reason captures two kinds of internal activity: motivations associated with how to achieve specific goals or outcomes, and beliefs regarding what the agent infers to be the current state of affairs. In other words, an agent is a well-differentiated system that acts based on beliefs and desires. This view is compatible with perspectives that have been developed by various disciplines: Behavioural science, which sees agency as goal-directed behaviour. Economics, which treats agency as the ability to select policies to achieve an objective. Cybernetics, which conceptualises agency as the ability to regulate the environment and keep it within a desired subset of states. Instead of taking the working definition[2] agency = beliefs + desires at face value, in this post I will discuss conceptual tools that can allow us to derive this construction from formal considerations. In particular, I will discuss the separation principle, which I believe has potential to provide a rigorous foundation for this conceptualisation. These ideas take inspiration from cybernetics and computational mechanics, but distinctly leverage classic results in optimal control theory and reinforcement learning. Motivation — cybernetics. A related line of thinking pertains to the (in)famous Good Regulator Theorem. This result, put forward by cyberneticists Conant and Ashby in 1970, suggests that an effective controller requires an internal model of what is being controlled. This makes intuitive sense: it is hard to predict a system that follows an intricate internal mechanism, and perhaps the only way to effectively control it is by understanding it. Unfortunately, the original paper has been both a source of inspiration and confusion. Lots have been written trying to clarify it; to read more, see this post and this post, and also this manuscript. This post will follow these ideas in spirit, but not the formalisation as proposed there. Motivation — computational mechanics. Another related line of work is computational mechanics, which combines principles of information theory, theoretical computer science, and some sprinkles of statistical mechanics to describe the dynamics of prediction. Computational mechanics derives minimal structures required for optimal prediction, which take form in the mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[