Against strong decision-theoretic realism

N.B. Some of this post argues by analogy between decision theory and values. I expect at least these parts of the post to be unconvincing to anyone who expects sufficiently smart agents to converge on the same values, as some moral realists do. I will not argue against moral realism here (see e.g. this sequence for one such argument).

Note that I take a convergence-based definition of realism for this post. One could hold that there is a truth about the correct decision theory, but that agents won't necessarily converge on it. I don't argue against views like that here. I am interested more in the question of convergence than of truth, because the former bears on whether ASI's decision theory is path dependent. In an upcoming post, I will argue further for path dependence, and for the time sensitivity of interventions to influence AI's decision theory. This is the third post in our sequence Intro to acausal interactions.

In this post, I argue against the following claim, which I call strong decision-theoretic realism: that sufficiently smart agents will all converge on the "correct" decision theory (DT). In doing so, I also argue against a related claim: that absent "lock-in", humans together with somewhat aligned AIs will necessarily converge on a reasonable decision theory.

As a consequence of this, I hope to convince the reader that the avoidance of "lock-in" should not be the primary focus when considering decision-theoretic reflection processes.[1] Rather, we should care about the overall quality of the decision-theoretic reflection process (where quality is defined with respect to our object- and meta-level decision-theoretic commitments). As with values, there are certain things that we want to lock in (e.g., fundamental intuitions, some properties regarding how we wish to reflect), and other things that we don't (e.g., complicated object-level properties that we are relatively uncertain about). There are many possible "reflection processes", but when we say we want to reflect on our values (or DT), we have in mind a relatively specific subset of these. For instance, we are probably generally pretty happy for the process to involve reading philosophical arguments, and probably less happy for the process to involve taking a litany of mind-altering drugs. Whether the default process will be satisfactory therefore depends both on empirical facts (how reflection will proceed by default) and on our own meta-decision-theory (how we wish reflection would proceed). Were DT realism true, it would imply that a very wide range of reflection processes would lead to good outcomes, and so we would need only avoid very bad processes, like locking in a particular decision theory prematurely.

First, an argument against the strongest forms of DT realism: A true CDT agent and an EDT agent will never reach the same decision theory, no matter how many arguments they are exposed to. "Why Ain'cha Rich?"-style arguments don't work within the framework of CDT (see Caspar Oesterheld's post on the lack of DT performance metrics, and e.g. Bales (2018) and Wells (2019)).

While CDT does recommend self-modifying to "son-of-CDT", the gap between son-of-CDT and acausal decision theories is large. Roughly speaking, CDT wants to commit to the CDT ex ante optimal policy at the point it is able to do so. Thus, son-of-CDT acts in an updateless way with respect to things that are causally downstream of its decision to commit, but not with respect to other things (e.g., events that happened before its decision to commit). This leaves it very far from updateless EDT, or even causalist updateless decision theories (such as FDT), because most of what matters isn't causally downstream of any decision an agent ever makes after existing.[2] It will not do ECL with aliens, or even with copies of itself in branches that split off before it self-modified, but only with copies in branches that split off afterwards. Where predictions of it were made before it self-modified, it will still two-box in Newcomb's problem, and lose money in expectation when faced with the Adversarial Offer[3] (Oesterheld and Conitzer, 2021). EDT likewise recommends self-modifying to "son-of-EDT", which is more updateless but not any more causalist. Something similar holds for agents who hedge between decision theories (depending on how the hedging is operationalised).[4]

Of course, we can still imagine agents whose decision theory is CDT-like (say) for most object-level purposes, but who have a meta-level commitment to some kind of reflection procedure. Perhaps we would characterise lacking such a meta-level commitment as a form of lock-in, in which case this point doesn't, by itself, argue against the idea that all we need is to avoid "lock-in". The following sections argue that avoiding lock-in in this sense still wouldn't guarantee convergence.

We have no theories of decision-theoretic reflection that rule out path dependence. This contrasts with theories of updating on empirical information. We therefore lack reason to think that path dependence and failure to converge are purely a result of human bias, and that more idealised agents would avoid this.

We have good (and comparatively uncontroversial) theories of how agents should update on new empirical information.[5] In particular, under Bayesian updating, one's ultimate credence does not depend on the order in which information is received (either way, one ends up multiplying one's prior odds by the likelihood ratio of all the data). Agents with the same prior should agree, given the same evidence, and even agents with different priors should approximately agree, given enough evidence (provided neither prior rules out the truth). Thus, when we see humans failing to converge on empirical matters, we can say that either they haven't spent enough time sharing evidence, or they are acting irrationally, perhaps due to confirmation bias. We therefore have reason to think that smarter agents[6] would eventually agree on empirical questions. It seems like something similar should hold for logical questions — at the very least, once something has been proven, all agents should agree.[7]

The same is not true of decision-theoretic reflection (or moral reflection). We lack any theory of such reflection, let alone one that implies path independence, or convergence more broadly. Moreover, updating one's decision-theoretic beliefs (or values) typically fundamentally changes how one makes future decisions, including potentially how one evaluates further arguments, so the order in which one encounters arguments can matter.

Decision theory disagreements often reach a point where the remaining disagreement lies in the realm of raw intuition, and where further progress is elusive. This seems to pose a challenge for the possibility of eliminating disagreements by sufficient reflection. It also mirrors the behaviour of discussions of values.

Persistent disagreement in and of itself isn't necessarily strong evidence of antirealism, because there are often alternative, equally good, explanations of the disagreement. Therefore, to assess whether persistent disagreement is strong evidence of antirealism, one has to consider how discussions on the topic actually proceed, and how well other things would explain the disagreement.

In many domains, disagreements persist because they are a feature of complex world models, and mapping out the disagreement and surfacing cruxes takes a lot of time and energy. Thus, even if there is a truth of the matter that everyone would eventually converge on, one would not necessarily expect disagreements to be resolved quickly. For instance, I think this is the case with a lot of AI risk discussions. By contrast, decision-theoretic disagreements (and ethical disagreements) appear even in very simple thought experiments. Moreover, such disagreements seem to reach argumentative "bedrock" relatively quickly compared to other questions. Take the e