[Diagram] Early handoff? Improve conceptual reasoning?

This article is about: How do we (more) safely defer to AIs? (Ryan Greenblatt, Julian Stastny) AI 2040: Plan A, Alignment Roadmap (Ryan Greenblatt, Thomas Larsen). If you've read them, I'm impressed, they're both very long. If you haven't read them, you might be confused about: Why should we "hand off" to early AIs? Shouldn't we use control? How does improving AI's conceptual reasoning reduce overall risk? Won't this make them better schemers? For the sake of my fellow Greenblattologists, I have tried to boil down the arguments to a simple diagram. Motivating scenario. Responsible Leader. Let's assume that we're advising a reasonable AI company, with a 1-12 month lead over its competitors. The company will have poor incentives, it's managed by humans with typical flaws. However, the company has broadly good intentions, and isn't wildly mistaken about the strategic situation. Conceptual workload. The reasonable AI company faces exogenous risks, e.g. a reckless competitor, or a rogue misaligned AI about to hit a software-only singularity. Managing these exogenous risks would require a sizable load of conceptual work, which is fuzzy, philosophically-loaded, and hard-to-verify. This includes: Evaluating the risks of current deployment; threat modelling and designing safeguards; general science of AI; training future AIs that can be trusted Negotiating with the competitor; handling domestic and foreign governments, the public; legal and regulatory risks Keeping humans "clued up" on the situation; knowing whose advice to trust Exotics: acausal stuff, superpersuasion, space governance, AI welfare There aren't enough humans around to do this in time. The AIs seem capable and aligned. The company has AIs which look capable, aligned, and well-elicited on easy-to-verify tasks like programming, proving theorems, ML research with short feedback loops, etc. Behavioural testing hasn't found egregious misalignment, so the remaining possibilities are: The AIs are scheming (covertly egregiously misaligned) and have evaded our testing. The AIs are broadly aligned, i.e. they might occasionally do weird misaligned things, but rarely enough not to hinder our ability to safely deploy them. These might be residual drives from RL or persona selection, which occur on out-of-distribution contexts. In Greenblatt's taxonomy, the live possibilities are Slopolis, Lurkville, or Easyland — not Hackistan or Schemeria.[1] Our optimisation problem. We're advising the company on how to minimise total risk. Simplifying somewhat, this is the sum of two things: Exogenous risk. The chance that the rival causes a catastrophe. Loss-of-control. The chance that the responsible company deploys covert schemers who subvert the oversight and cause catastrophe.[2] The company's strategy consists of three phases: Continue training bigger models with greater capabilities. Pause at some capability, then ensure the AIs aren't scheming. Evaluations, red-teaming (and red-teaming the red-teaming), honeypotting, offering bounties to reveal misalignment, model psychology, general science, internals stuff, measuring performance on proxies of the deployment, observing how its week-long plans turn out, etc. They'll be finetuning different versions of the model, guided by these evaluations, hopefully without overfitting. Deploy the AIs to manage the exogenous risks, via some oversight protocol. The AIs might build successor AIs which are (hopefully also) competent and aligned. Overall, company's strategy is characterised by three levers: Train-Lead ratio. How much of the lead time should the company spend scaling to larger models, before pausing for evaluation? Eval-Deploy split. What fraction of the remaining time should the company spend evaluating the model, before deploying? Oversight. When the company deploys the AIs, how much should humans check the AI’s thought processes, output, etc? Let's walk through how these levers determine the overall risk. Overall capabilities. The longer the company spends scaling to larger models, the more capable the models will be. This includes: Conceptual capabilities required for managing the exogenous risks. Think: strategy, epistemics (forecasting, threat modelling, philosophy, introspection), ML research taste (so the AI doesn't lose the race). Scheming capabilities required to (1) undermine our evaluations and (2) subvert our oversight. Think: opaque reasoning, controlling chains-of-thought, controlling internal activations, steganography, collusion, eval-awareness, sandbagging, distinguishing honeypots from real opportunities to attack. Note that scaling to larger models also reduces the remaining lead time that the company can spend on evaluation and deployment. At the limit, the company could spend their entire lead on building the most capable model they can. This would have the maximal conceptual abilities, but the AIs would have little time to actually deal with the exogenous risks. Scheming propensiti