Utilities as Legendre duals of probabilities

TLDR: In recent work, Roy Fox proposes to understand an agent's capabilities in terms of the set of environment dynamics it can bring about.[1] This leads to an intriguing duality between probabilities and utilities via the Legendre-Fenchel transform. Introduction Some agents are more powerful than others. Indeed, some can yield a wider range of outcomes, maybe because they are capable long-term planners or because they have built rich world models. Being able to clearly delineate the capabilities of agents is an important challenge for AI alignment. A natural place to start thinking about how to describe the capabilities of an agent is reinforcement learning (RL), or more generally, approaches that see behaviour as arising from the maximisation of expected utility. By taking this view, one can describe "capability" as the range of reward/utility functions that an agent can successfully maximise — as done e.g. in classic work by Legg & Hutter and also in more recent work. Such a perspective is very useful, but I am not a big fan of rewards/utilities. Rewards are great in games and other settings where they come naturally, but real life often does not handle rewards on a silver plate. When absent, rewards are often defined artificially — but the design of rewards that give rise to desired behaviour without enabling Goodharting is extremely difficult. That said, reward-based frameworks are powerful,[2] and proposing alternative approaches that can compete with them is hard.[3] Thus, while I am not a utility/reward fan, I am intrigued by situations where they arise naturally. One place where this happens is in the celebrated von Neumann-Morgenstern utility theorem (VNM), which states that preferences with specific properties can be described as if the agent is trying to maximise a specific utility function (see this great post for related discussions). Another interesting idea is the link between utility maximisation and description length minimisation presented by John Wentworth, which suggests a duality between utilities and probabilities.[4] A related perspective appears in Jeffrey–Bolker rotations as discussed by Abram Demski, where probability and value form vectors and certain linear transformations can move structure between them without changing the represented preferences. I recently stumbled upon yet another way to see how utilities can naturally arise from probabilities, this time via the Legendre transform. In contrast with the settings discussed by VNM, Wentworth, and Demski, which assume a given preference or utility, here we simply start from achievable environmental dynamics to later derive utilities from them. The idea is simple and elegant — I'd not be surprised if it has been discussed before.[5] Moreover, I believe this perspective may be useful for various issues related to AI alignment, which is what pushed me to write about it here. The rest of this post presents: A general way to describe the "capability space" of an agent. This description uses probabilities and policies, but doesn't use utilities or rewards. An explanation of how this definition affords a dual description via the Legendre transform, which naturally establishes utilities as Legendre duals of probabilities. Defining capability space Let's consider an agent that acts over an environment described by a variable mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block;