Two Paths from Logical Induction: Part 0

Written as part of the MATS 9.1 extension program, mentored by Richard Ngo. Additional thanks to Maria Kostylew, Lorxus, and others for helpful feedback on this post. Introduction In this sequence, I will outline two prospective agendas that build on Logical Induction (LI) to contribute to a descriptive theory of embedded agency. In part 0, I introduce LI and cover some progress the framework has made towards clarifying key properties of intelligence. This piece then concludes with a brief case for both the research directions. I am optimistic that these lines of inquiry could shed light on some fundamental concepts such as actions, inductive biases, and preferences. 1: Why Logical Induction? 1.0: A short introduction Logical Induction is a 2016 paper that provides a theoretical proof-of-concept for logically uncertain bounded reasoning. Its key motivation is that bounded reasoners cannot fully propagate the consequences of their beliefs. They contrast (for example) Bayesian Networks (BNs), a standard mathematical framework for belief management. The idea of BNs is for dependencies between probability distributions to be represented by a Directed Acyclic Graph (DAG); updates to a set of distributions spread throughout the graph via a formal Bayesian update rule. Bayesianism has some philosophical issues relating to unknown unknowns that we'll get to later, but their prosaic obstacle is that fully propagating updates (known as inference) is NP-hard. Therefore, understanding bounded cognition partially comes down to explaining how epistemic systems can perform computationally limited, incomplete belief updates while still achieving (weakened) rationality conditions. Logical Induction addresses these issues in a setting where a reasoner cannot always figure out the implications of their beliefs as described in mathematical logic. The setup is a sequence of reasoners that, at each index n[1], assigns credences between 0 and 1 to logical statements. The function mapping statements to credences is called a belief state. These belief states are simultaneously interpreted as markets that set prices in these statements. Traders are sequences of programs that, at each index n, issue buy and sell orders in logical statements according to the prices set by the market. The arbiter of truth is a nested sequence of finite sets of sentences, called the deductive process, that represents statements that are revealed, observed, or proved[2] to be true up to a fixed time-step. An issued share in a sentence cashes out at $1 at the end of time if the sentence is plausibly[3] consistent with the deductive process. Traders are rewarded in the system's internal currency for behaviours such as: buying shares in true statements before they are shown (or determined by consensus) to be true selling shares in false statements before they are shown to be false anticipating market price shifts to buy or sell shares in sentences pre-emptively. A trader is said to exploit a sequence of belief states if its plausible wealth at the end of time is bounded from below, but not from above. Such traders are intuited as conducting arbitrage on the market prices, making risk-free unlimited profits. The rationality criterion for a sequence of reasoners (or, equivalently, for their belief states) is that no trader exploits it. This is a weakened anti-Dutch book or coherence condition. Indeed, the authors go on to prove that "inductors" that satisfy this criterion have various qualities (in the limit of infinite time-steps) one would expect of a consistent, accurate reasoner. For example, inductors learn to have stable beliefs. They also learn to assign provable statements[4] a probability of 1, and are well-calibrated in their beliefs. I'll now zoom in on two aspects of Logical Inductors that are foundational for the directions covered in the sequence: introspection and inductive biases. 1.1: Introspection and game theory One of the more remarkable properties of inductors is the ability to introspect. Classical reasoning systems struggle to coherently evaluate statements about themselves. These difficulties stem from paradoxes of self-reference where the truth of a statement depends on the reasoner's belief about that statement. A simple example is mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-con