Sequent: scale and automation for higher confidence in alignment

Alignment is not on track Artificial superintelligence (ASI) may be developed in the next few years. It is unclear whether alignment is on track to be ready on the same timeframe. At a minimum, the empirical programs at AI labs are unlikely to deliver a priori confidence, before training ASI, that things will go well. We are starting a large nonprofit research organization, Sequent, that aims to clear a higher bar: We are aiming at higher confidence via a portfolio of theory and empirics bets, all of which could fail, such that if any succeed, they would give us more a priori confidence in aligned outcomes. We are investing heavily in automation to accelerate progress on these bets. We believe that theory unlocks higher automation. Taking a more principled approach offers better filters for deciding which directions of automated research are promising (a proof is worth a thousand experiments, and even a pseudo-proof is worth hundreds). Who[1]: researchers from the UK AISI’s Alignment Team and Timaeus, with more to come. We’re aiming at 40-80 FTE two years from now. The Alignment Team ran the £30m Alignment Project, and Timaeus has pioneered applying singular learning theory (SLT) to alignment. Founding team: Geoffrey Irving — Chief Scientist at UK AISI; ex-DeepMind, OpenAI, and Google Brain. Daniel Murfet — Head of Research at Timaeus; left tenure to pioneer SLT for alignment.  AISI Alignment — Alex Holness-Tofts and Jacob Pfau. Timaeus — Jesse Hoogland, Stan van Wingerden, and Marco Cozzi. Joined by researchers from Timaeus and more researchers from the UK AISI’s Alignment Team Where: a large in-person presence in the Bay Area (Berkeley), as well as researchers working remotely from London, Melbourne, and elsewhere. In this post, we discuss: What it means to aim at higher confidence Why start a new big organization Whether sufficiently fast progress is possible with automated research Aiming at higher confidence In an ideal world, we would develop an approach to building superintelligence together with a theoretical proof that it was safe, and then build it. In this world, we probably have to settle well short of this ideal. However, we believe this doesn’t mean giving up on the idea of leveraging theory to get safety. Our approach to alignment is differentiated from that of the AI labs due to our emphasis on seeking principled reasons for being confident that the alignment we observe in situations we control (for example, in training, or during evaluations in chosen environments) generalizes to alignment in situations we cannot easily control (e.g., large-scale, long-horizon tasks executed in the world). Most AI lab approaches are essentially reactive, resulting in methods that, while functional, do not yield principled insight into if or when they will fail. The “reasons” we have in mind range from a better scientific understanding of the underlying phenomena of deep learning to asymptotic guarantees of safety under specific protocols: Understanding deep learning is an unfinished scientific project. It is worth noticing that if we had a very complete theory of how deep learning works, certain kinds of alignment problems would be trivially solved (e.g., if we could rule out certain kinds of outcomes using an understanding of the training data and optimization process). We are not aiming at a complete understanding, but we think it is important to aim at a better understanding. This can be turned into improvements in how alignment works at the frontier in the near term (e.g., through changes to post-training). An asymptotic guarantee provides assurance that an AI training protocol (e.g., scalable oversight via debate) converges to a tolerably low chance of unsafe behavior, provided some parameters (e.g., training time and amount of data) are made sufficiently large. Such guarantees exist in the theory literature for some reinforcement learning algorithms, but not for neural networks or other classes of learners sufficiently powerful to plausibly reach AGI or ASI. We are not aiming to prove such guarantees for neural networks. We do however aim to prove asymptotic guarantees for alignment while making certain assumptions about the training process. This dovetails with an empirical effort to shift the practice of alignment as close as possible to an approach where we get confidence from these guarantees (even if the hypotheses of the theorems don’t strictly hold). However, we do not know any attacks of either form that are highly likely to succeed individually. Our hope is to build towards that confidence by exploring many different research bets in parallel, using a single organization setting to increase both sharing and amortization of work. Why a new big organization There are a number of existing non-profit organizations pursuing theoretical aspects of AI alignment. However, none of these have yet succeeded in making the transition to affecting how deep learning works at frontier scale, and the