[Macroagents] 2. Design lenses for optimizing macroagents

Follow-up to: The Macroagent Ontology (especially section 1.2 is a prerequisite) The previous post laid out the basic macroagent ontology. This post will look at some further concepts that are important lenses for optimizing macroagents, along with some ideas for each lens. We usually want to shape macroagents so that (1) their optimization target is aligned, (2) their epistemics are good, and sometimes (like for non-evil human macroagents) so that (3) their efficiency is high. I will call those our 3 target dimensions. I will go through 5 design lenses which are useful for optimizing macroagents: Incentives (affects all target dimensions) Macroagent evolution (affects all target dimensions) Information and decision routing (affects mostly efficiency) Knowledge aggregation (affects mostly epistemics) Value aggregation (affects mostly alignment) (These lenses focus on the macroagent structure and not on improving competence of agentic subsystems, even though in principle you could count things like project management protocols as mechanisms.) (The last two lenses apply more narrowly — only to macroagents that explicitly aggregate beliefs or values — but they're the main levers for epistemics and alignment specifically.) (TODO think about how MECE that is and maybe make comment on it. How well can other design lenses like management principles be represented here?) 2.1 Optimizing Incentives 2.1.1 Incentive failure modes Incentive failures arise when agents doing the work don't benefit proportionally to how well the work fulfills its purpose. (This is especially common if decision makers are not beneficiaries.) As an example, the purpose of science is (to a large extent) to make advances which enable the creation of technologies and methods that improve human well-being, but the daily incentives a scientist in academia faces are only a very crude proxy of this. There are two central blockers that make it hard to avoid incentive failures: Whether the work fulfills its purpose well is hard to measure. Bad incentives exist as part of a Nash equilibrium, so permanently improving incentives, i.e. getting to a better stable equilibrium, often requires changing large parts of the system they are embedded in. (Those difficulties can coexist in the same system.) 2.1.1.1 The hard-to-measure problem If you have a hard-to-measure task, you have 2 options: You just tell the agents to try doing the task without grading them, in which case you get slop. You try to find a decent proxy to grade them anyway, in which case you get goodharting. Slop. If you're on a job where there are some (measurable) tasks that you're being in some way graded on (e.g. your performance here affects what your boss thinks of you), and some hard-to-measure tasks where your results don't have consequences for you, you're incentivized to slack off on the hard-to-measure tasks and put more time and effort into the measurable tasks so you get promoted faster.[1]Even if there aren't other measurable tasks you could optimize, humans are usually still more motivated to try hard if they (strongly) benefit personally from good performance, and AIs often face elicitation problems and don't really know how to try hard or sensibly on tasks they haven't been RL-trained on. (Btw, the tradeoff of how strongly to reward measurable tasks when there are also hard-to-measure tasks is formalized in the Holmström-Milgrom multitask model. One takeaway is that it can be better to deliberately weaken incentives on the measurable tasks, or to give the measurable and the hard-to-measure tasks to different agents.) Goodharting. One common example of goodharting is students optimizing for exam performance instead of useful long-term learning. Hard-to-measure dimensions like depth of understanding and long-timescale model-building abilities are learned less. Cheating on tests (or, in the case of AIs, reward hacking) is also a kind of goodharting. The hard-to-measure problem is not only a problem for what agents are motivated to do, but also for what they are capable of doing. Both of these problems are probably significantly worse for AIs than for humans. On the motivation side, humans have social instincts that make them non-sociopathic, whereas with LLMs we are already seeing stuff like the Hugging Face incident (and such stuff will become harder to avoid as we do more RL training). On the capability side, (very smart) humans are sometimes surprisingly good at hard-to-measure tasks, e.g. Einstein making progress towards general relativity, whereas LLMs are likely differentially (much) worse at such stuff. 2.1.1.2 Bad Nash Equilibria Moloch's toolbox is a fantastic post that explains how many systems in our society are stuck in bad Nash equilibria. To just show one example here: cecie:  I’ll now introduce the concept of a signaling equilibrium. To paraphrase a commenter on Slate Star Codex: suppose that there’s a magical tower that only people with IQs of at least 100