The Open Problems of the AI Alignment Field and their Cruxes

Previous: AI Safety Interventions TL;DR: I made an overview of the open problems of AI alignment that reveals cruxes within those open problems and missed opportunities for formalization and collaboration. And CEV may deserve a second look. Epistemic status: Trying too much in too little time. I'm confident I have identified and modeled significant structure within the alignment field, but I urgently need feedback on specific gaps and this post is largely a call for that. My work was LLM-assisted, but no part of this post was LLM-written, except for the crux summary and the Lean code. Recently, Chi Nguyen and peterbarnett said: PSA: Almost nobody is directly working on superintelligent alignment. I have been around in the field since the old days of LW 1.0 and thought: that can't be true. I mean, so many people seem to be working on it. I thought I was working on it. But was I? The PSA made me think back on what I was actually working on. It was Steven Byrnes who came up with a research agenda I could actually contribute to, which led me to founding project aintelope in 2022 (PS. It is still going). And a while back, I compiled a report on AI Safety Interventions for AE Studio, so I thought I was kind of up-to-date on research agendas. Now, is that superintelligence alignment? Or is that more prosaic normal alignment? Maybe each individual research is normal, boring, but does all of that research together actually cover all areas? So I set out to collect the evidence and find and piece all the research agendas together, and connect to all of our field's open problems. This involved finding all the agendas, naming the open problems, and finding the evidence of each agenda for each problem. The first was easy, thanks to the people who maintain sites like aisafety.com (thanks guys for the great map!), my previous work, and other such resources (and aided by LLM search). The second was the hardest, as it involved getting a good understanding of the cruxes within the open problems and how they relate. That took most of the time. I am not reporting on all the details here; I owe you more posts on that. But once the modeling and structuring was done, putting it all together was again easy. The result is the pretty big table below (full clickable version can be found on my website, the tags classify evidence: C conceptual, T theory, S simulation, P practical, D empirical (software), E empirical (other), O other): This table shows some interesting patterns: Many cells are empty. In fact, some columns are almost empty. Some columns are crowded. Some columns have dominant types of evidence. There is little practical or applied evidence ("P") that I could find. And I think you can spot more patterns. Each pattern can have multiple explanations. For example, a group of empty cells can mean that some areas are genuinely neglected and more people should work on them. It can mean an open problem or crux was dissolved, i.e., there is actually no longer a need to work on it, or it isn't load-bearing anymore. Or it could be a coding error on my side. The open problems Value Referents (MB3, part of Value Learning), Tiling (MB5), Acausal Coordination (MB7d), and CEV (MB8) seem to have little evidence. Is that because matching them with existing research is hard? Due to a lack of shared terminology? Or is it because of errors in my modeling? Relatedly, the last row, my work, Towards Superintelligence Alignment (TSA), appears to span everything, but that is simply because I know how my work relates, and can easily link it. Also, it is largely of a different type than the other evidence - no published papers but mostly simulations. So don't treat that as comprehensive coverage but as advertisement. But it reminds that visibility of agendas and publication matter for the other cells too. The matrix doesn't tell why cells are empty and why the patterns are there, but the answer seems important. It is clear that alignment must cover the columns against a superintelligent adversary. But the matrix also doesn't tell us yet if each cell delivers. The second thing I want to elaborate on a bit more is some structure within the open problems. As far as I can ascertain, the open problems are Embedded Agency, Value Learning, Corrigibility, Tiling, Goodhart Selection (the principle behind Gradual Disempowerment), Inner Alignment, Acausal Coordination, Extrapolated Volition, Grounding Drift, Successor Gaming, and Deployment Safety (the crucial part of AI Control). LLM assisted with Cursor Composer Fast: Summary of open problems from my site (lightly edited) MB1 — Embedded Agency A measured agent–environment cut is sound enough that the certified unit is the real control locus. The embedded-agency problem denies a clean Cartesian cut—the real optimizer may not be the visible model. We can treat the boundary as a measurable object, then the bet is on estimator soundness, not absence of cuts. Related: Boundaries, Part 1, directed Markov