Price recursion is the rational theory of reward

(The original title for this was "Markets are equivalent to stuff", because I also demonstrate market-MDP/Bellman and market-backprop analogies. While those scratched an itch I've long had, the more important result is the headline one. Part of my work at MATS in Richard Ngo's stream.) setup and price recursion/backprop Markets are MDPs Chain markets are MLPs Continuous Bayesian inference as a dynamics for markets Markets : Reward = Bayes : Beliefs (Markets rationalize reward and identity) appendix: Caveats and questions A bunch of people have vaguely noticed intuitive analogies between economies/markets and machine learning algorithms: John Wentworth, Aanjaneya Kumar and Anders Sandberg at the AI Objectives institute, the ancient Neural Bucket Brigade (Davis, Schmidhuber). Several papers discuss RL algorithms based on sequentially auctioning a unit "right to act" among agents---most recently Caspar Oesterheld's BRIA but also Baum's Hayek Machine, its predecessors the classifier systems (Holland 1985, Schmidhuber 1987, Schmidhuber 1989) and its successors Hayek-for-POMDPs (Kwee et al 2001) and Hayek with Vickrey auctions (Chang et al 2020). In "Market-based architectures in RL and beyond" I suggested that a better market model for RL might come out of factoring the "right to act" into "goods" axes, just like in real markets. The analogy is attractive and suggestive: Economies are intelligent agents built out of sub-agents much less capable than themselves (see e.g. "I, Pencil") suggesting that markets are an example of intelligence being an "emergent phenomenon". Bounded rationality is very analogous to the Efficient Market Hypothesis. How can you be rational when computing the rational decision itself has a cost---how do you optimize over that? You are rational/markets are efficient "conditional on the information (including computational information) present in the system", whatever that means. Markets have analogs of inconsistency (arbitrage) and Godel's incompleteness theorem: a prediction market is limited in its ability to efficiently price the question "This question will resolve YES if this market gives less than 50% probability to it". (More specifically, a market must give zero subsidy to such a question: its subjective "value for information" on that question must be zero.) Related to 3: "Market dynamics" is famously under-studied because again it is fundamentally computational and boundedly-rational rather than "there exists an equilibrium, DONE". The dynamical nature of markets feels very similar to backpropagation (as well as e.g. Bellman recursion): prices are gradients (specifically: gradients on activations, not weights---they do not zero out at equilibrium), the gradient In this article we will formalize this intuition, and explain that in fact, markets are equivalent to a lot of things: backprop, RL, and continuous Bayesian inference (aka "replicators"). In the first section we will describe the general economic model and how "prices" (gradients) arise naturally. In the subsequent sections we will start describing the analogies. 1. Setup Here is how we will define an "economy". Definition 1. There is a vector space mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; b