Explaining Knightianism on one foot

I’ve tried several times to summarize the core question my research is trying to tackle (and, indeed, I often think of research progress as a process of asking increasingly good core questions).[1] This post gives the deepest version of that question I’ve found thus far: how should you relate to the parts of the world you can’t directly model or control? Let me explain further in terms of a distinction between two perspectives. From the third person perspective you think of yourself as “outside” the world, looking in. You’re a good Bayesian, in that you have a set of mutually exclusive collectively exhaustive hypotheses. You choose actions by multiplying your credences by your utilities over those hypotheses, and you treat those actions as the only way you influence the world. Some problems with the third person perspective (aka Cartesian or dualistic agency) were described in Scott and Abram’s sequence on embedded agency. One crucial issue is that most realistic environments contain other agents which are modeling you back, which means that your thoughts might affect the world via channels that aren’t just your actions. Game theory somewhat mitigates this problem, but only in the very specific case where all agents know (that all agents know, that all agents know…) that they’re in an equilibrium. A more subtle problem is that, even in the absence of other agents, modeling yourself as part of your environment can raise contradictions when you consider taking different possible actions. Another key problem is that any realistic agent is too “small” to model most of the wider world. If we take this problem seriously enough, we end up at what I’ll call the first person perspective. From this perspective, the world is primarily a stream of sensory data—you don’t have a set of hypotheses that reliably carve it into mutually exclusive (let alone collectively exhaustive) possible worlds. Your main job is to learn (partial, overlapping) concepts that help to predict and explain that data. Predictive processing is one framework which implicitly operates from the first-person perspective (though the full active inference framework is a bit more complicated); so does perceptual control theory. You can think of babies as operating from the first-person perspective; I also find it helpful to think about it as the perspective of individual cells. Each cell has some boundary with the rest of the world, through which it accepts sensory inputs, and outputs externally-facing actions. It has some primitive proto-concepts (like light, dark, food, danger) which it uses to guide its actions. A cell which is inside a friendly body should be more open to its environment; a free-floating bacterium should be more cautious (h/t Sahil for this point; Michael Levin also has some very interesting work on cell-level agency). This intuition pump may seem extreme, but there’s an important sense in which we’re all much closer to the cell’s-eye perspective than the god’s-eye perspective. Our concepts are radically incomplete; we’re deeply confused about how to carve the world up—and this might not change even in the limit of increasing intelligence, if other agents are also becoming more intelligent at the same time. The big question in my mind is how to synthesize the two perspectives into a theory of (non-probabilistic) uncertainty about possibilities that you can't fully represent—which I've been calling a theory of Knightian uncertainty.[2] Such a theory would acknowledge unknown unknowns (as in the first-perspective), but also give you a principled way of dealing with them (as in the third-person perspective). One intuitive picture I’ve been using: we can move towards a theory of Knightian uncertainty by considering Bayesian hypotheses with “holes” in them corresponding to “Knightian regions” which we can’t model or control (e.g. other agents smarter than us). You can also think about the first-person perspective as coming from “inside” one of those holes, looking out. Ultimately, I expect that this will end up resembling a Sierpinski triangle where, the further you zoom in, the more holes there are; I also expect that different regions will have different levels of “Knightian-ness”. But even this simplified picture already gives us some useful intuitions—for example, when there’s something we can’t model or control, we want to quarantine it so the uncertainty doesn’t “infect” the rest of our world-model. However, adopting a fully adversarial stance towards Knightian regions (which is my rough understanding of what infra-Bayesianism does) seems extremely costly. Increasingly, I’ve come to suspect that a theory of Knightian uncertainty will bridge the first- and the third-person perspectives by adopting a second-person perspective—i.e. by taking a relational stance towards each Knightian region. By “relational stance” I mean something like “choosing how deeply to entangle your beliefs and actions with what’s happening in that region, bas