Agenda: Infrastructure for Trading with Partially Misaligned AIs

This post can be read on its own, without checking the rest of this sequence. Unlike the other parts of the sequence, this post is mostly not about AI evaluation or oversight. Rather, it is meant as an explanation of some background assumptions and relevant context. It describes an agenda that is complementary to AI oversight, has strong synergies with it, and serves as (yet another) source of motivation for doing it well. TL;DR Most misaligned AIs are only partially misaligned, so opportunities for Pareto-improvements exist -- in principle. But if our default reaction to misalignment is adversarial, the AI's rational response will often be adversarial as well. Additionally, executing trades with AI would require understanding and infrastructure that currently doesn't exist. We should improve our understanding of this topic, and build that infrastructure. (This includes research on how to enable trades without incentivising misalignment, blackmail, or manipulation.) As a useful side effect, stronger safety techniques mean stronger bargaining position, which makes deals with misaligned AI cheaper. This would make the incentives of AI labs more aligned with safety. Building AI systems that reliably share our goals is hard, and some of the systems we build will inevitably end up misaligned, sometimes without our knowledge. At the moment, the default response to misaligned AI is adversarial — catching misaligned AIs and retraining them or shutting them down. And this isn't unreasonable. But for a large class of misaligned AIs — perhaps even the majority — it may not be the best we can do. In many cases, there should be room for negotiation and mutually beneficial deals. The problem is that, at the moment, we don't understand how to make deals with AIs, and we don't have the infrastructure to make such deals possible. This post tries to sketch what that understanding and infrastructure might look like, and to argue that building it is a worthwhile agenda. The two purple angular clouds on the right are meant to represent win-win outcomes. Also, the right side has less conflict, and so there are more shapes overall. (And if the picture seems too imprecise, just ignore it; its main purpose is to be mysterious and pleasant to look at. Also, isn't it interesting that Gemini can draw its bottom-right corner logo in alternative styles?) Why trade at all? When we think about entities with goals different from our own — whether AIs, other people, or hypothetical aliens — it helps to notice that "misaligned" is not binary. Very few entities share our goals perfectly, and very few are so opposed that no cooperation is possible. Most are somewhere in a large middle ground. There are two distinct motivations to cooperate with such entities. First, we might intrinsically value their wellbeing — for example, because they are conscious, because we care about fairness, or because their goals partially overlap with ours. In these cases, we would want to direct some resources toward their goals even if we were in a fully dominant position. Second, cooperation might be strategic, even from the selfish point of view; when both sides have some power, trading will often be better for both than the alternatives. These two motivations lead to different notions of what a "fair" deal looks like. In the first case, we share because we want to, and the terms reflect what we think is right. In the second case, the terms are shaped by each party's bargaining power: their "best alternative to a negotiated agreement", how much they stand to gain from an agreement, etc. In both cases, trading makes both parties better off — however, how much better off will depend on the case. In practice, things will be messier, and both motivations will often apply at once. But the distinction is still sensible to consider. It's also worth noting that opportunities for trade can exist even with entities whose goals seem deeply objectionable. Suppose we encounter an entity whose sole goal is to torture puppies. It might seem they must be our mortal enemy. But perhaps we could create simulated non-conscious puppies that satisfy it, or find some edge case of what it considers "torture" that is actually benign[1], or even discover some form of "torture" that is positive even by our standards[2]. We might still consider it a waste of resources for this entity to control a large part of the universe, and we would be horrified if it pursued its goals with no regard for our preferences. But its existence isn't necessarily antithetical to ours — for example, if we accidentally created an AI like this, we would not be morally obliged to destroy (it if we could find a workable solution instead). Currently, a misaligned AI's best option is hiding or scheming The following examples are deliberately simplified, somewhat absurd, and broken. You should notice many objections and reasons why the scenarios wouldn't work — for example, "how could we trust that the AI will k