Coalitional Darwinism and the Instrumental Utility of Individuality

This post was written as part of MATS 9.1 under the mentorship of Richard Ngo.

This post is the first of several I will be writing on using natural selection to understand artificial intelligence and agency. This post will show how noisy selection on genome structure can make evolution effectively non-myopic. From this, we give a Darwinian account of the emergence of 'individuals' constituted by coalitions of lower-level replicators.

Later posts will develop the connection between genome-structure selection and neural-network feature-learning, with applications to interpretability and alignment.

Note: I will use 'Darwinism', 'evolution' and '(natural) selection' interchangeably — more technical discussions of selection often distinguish between them.

Natural selection is limited by noise in its ability to resolve small differences in fitness. Because selection is a limited resource, and because selection makes lineages more fit on average, organisms will tend to evolve to make more efficient use of what selective power they have.

The noise floor can make evolution effectively hyperopic (the opposite of 'myopic') by buffering the effects of mutations which may be beneficial in the short-term but harmful in the long-term. The evolution of 'bet-hedging' is given as an example. Hyperopia permits the evolution of pre-commitments detrimental to some individuals.

The 'bowtie' network motif is an example where selectability is an essential property of the architecture. The network develops low-rank structure, limiting its expressivity, in order to be selectable. This may be understood as a 'coalition of the invisible' — network links which otherwise have too little effect to be tunable by selection instead commit themselves to this low rank structure, trading optimality for selectability. The coalition is then entrenched as selection optimizes it.

In prisoners' dilemmae, groups of organisms can evolve obligate co-operation (an inability to 'defect'). In some circumstances, obligate co-operators may evolve policing structures and joint heritability, resulting in the emergence of a new effective unit of selection. The suborganisms constitute the effective genome of the superorganism.

The noise floor limits how effectively subagents can be aligned to the superagent; this imperfect alignment provides the reservoir of variability which permits selection to act on the superagent. Thus subagent misalignment (which is typically only minimally deleterious) is synonymous with 'exploration' in the superagent's genome space. Incoherence (i.e., subagent conflict to the superagent's detriment) is the fitness subsidy paid by the individual to its lineage in the form of typically-slightly-harmful exploration.

1. Noise Limits the Precision of Selection

Summary: Due to randomness, natural selection can only reliably fix or purge mutations with large enough effects on fitness. This resolution is proportional to the (effective) population on which selection acts.

For clarity, we'll consider here species that reproduce in fixed generations, with constant fitnesses; the qualitative picture remains similar regardless. Spherical cow stuff.

Consider types of organism; at each generational turnover, they replicate according to a fitness , with the population of type at time . The fraction of the total population represented by type , obeys the replicator equation:

Where is the average fitness of the population at time , , which changes as the proportions change, even as we have assumed the fitnesses constant. This says that the proportion of the population made up by type grows in proportion to how much its fitness exceeds the population mean fitness.

1.A. Smaller Populations Have More Sampling Noise

In fact this is not exactly what happens — a type organism might perchance have slightly more or less than their allotted average number of descendants . One way to model this effect is to imagine the to be idealized probabilities, and, for a population assumed to have size [1] , we independently draw samples according to the categorical distribution on . I'll spare you some math (see, e.g., these notes).

In effect, this introduces noise with variance . Functionally, if two types have comparable populations and a fitness difference much smaller than the noise scale, the noise makes it impossible to resolve. Concretely, if all types have fitness , except for one type with fitness , selection will only 'see' this difference if

That is, selection has a resolution power proportional to the population size — larger populations mean the noise is averaged out. This is referred to as the drift-diffusion threshold — 'drift' means 'mostly governed by fitness differences', 'diffusion' means 'mostly governed by noise'. [2]

And of course, all of the above is a toy model — isn't exactly the real population, which should change over time anyways, etc. — the point is that noise is important and governs the resolution at which selection can operate.

1.B. Evolution is Usually Nearly Neutral

Evolution is solving a high-dimensional optimization problem, just like neural network training via backprop. In both cases, and I think this is probably a fairly general property of useful high-dimensional optimization algorithms, most directions in genome/parameter space are 'neutral' — they have effectively zero loss.

Two important upshots of this for now: first, if we look at an organism and see lots of complexity, it's not necessarily true that this complexity solves a problem as opposed to being a kludge. Second, this 'non-adaptive complexity' can still serve as the substrate for later adaptive mutations.

For example, in a regime where having long genomes is very cheap, it's mostly fine to have 4 or 5 copies of gene A. If, down the line, one has need of a variant of protein A, it's quite handy to be able to have those extra copies around so you can finetune the variant without messing up the funct