Perspectives in favour of improving conceptual reasoning capabilities

Below are some informal perspectives on improving AIs’ conceptual reasoning capabilities that motivate work in the area. Conceptual reasoning here means reasoning about questions where we cannot verify the answer, also don’t have data on unambiguously closely related questions where we can verify the answer, and need to rely on informal argumentation to make progress. This is a very quick post, so I’ll only gesture at each perspective without always giving a full justification. I’m also deliberately focusing on positive cases without going into potential objections and counter-objections in detail. Each perspective could have its own lengthy post. (Our team is still planning to release a more detailed post on the case for our work. I’m sharing some quick personal takes here.) 1: Moving from worlds that are not obviously bad to actually good I’m worried that, by default, a lot of safety efforts at labs will move us from a world where things are obviously bad to one where things are not obviously bad but still probably secretly bad, i.e., to the green trajectory below. (The graph is taken from a recent lightning talk on a different topic by Buck Shlegeris (CEO at Redwood Research, my employer).) In particular, I’m worried that we will do all the easy empirical safety where we hill-climb on safety evals until we no longer see any obvious, egregious misbehavior. But this doesn’t help us against subtler misalignment or misalignment that only shows under a distribution shift, for example from reward seekers who expect to be penalised for blatantly misaligned behaviour. In these worlds, easily attainable empirical signals of misalignment are scarce by design because we already optimised against them. Many others have already articulated this better and in more detail (for example, here, here, and indirectly in many of the pieces linked here). I think improved conceptual reasoning more or less directly targets moving from worlds that are not obviously bad to worlds that are actually good. (It also helps in obviously bad worlds but I somewhat expect that we’ll move away from those even without great conceptual reasoning.) With less empirical evidence available, we will need careful conceptual reasoning to assess how good our situation actually is, what could improve it, and what different pieces of evidence that we could collect would actually tell us. I think this is basically necessary for safety efforts to succeed robustly, and I would sure as hell like AIs to be able to help us with it. 1.1: Making good safety cheaper for AI companies The following perspective applies in general, but it makes especially good sense from the “conceptual reasoning moves us from not obviously bad to actually good” worldview. A lot of AI safety efforts aim to increase AI companies’ willingness to pay for safety. I think improving AI’s conceptual reasoning capabilities approaches the same issue from the other direction by making genuinely good safety research cheaper. As described in perspective 1, I think it’s easy for labs to default to doing very superficial, hill-climby empirical safety work, whereas I believe good safety work necessitates thinking about conceptual questions (e.g., non-technical aspects of “what does this experimental result tell me about the model’s out-of-distribution behaviour?”) with extreme care and rigor. But doing the latter is difficult and requires labs to pay costs for safety at a point when they could perhaps get away with doing less because their models no longer run around committing crimes, and misalignment is only arguable rather than blatantly obvious even to the public. Making models good enough at conceptual reasoning to do careful, rigorous and principled AI safety immediately puts large amounts of very fast, high-quality safety labour at the labs’ disposal. Perhaps this will make good safety work cheap enough for labs to actually do, especially if the AIs that they’ve used to help with the easy kind of safety are conceptually sharp enough to recognise that the situation is not actually good and shout at the labs about it. 1.2: Giving more sensible advice, increasing willingness to pay for safety Connecting to the last point, it just seems great for politicians, policymakers, journalists, the public and AI labs all to have access to better advisers on AI safety. It is much harder to pretend that your safety is fine if one of your main advisers keeps saying it isn’t. It would be great if models—which by default will probably be asked for their opinion a lot whether we like it or not—would be good, sensible such advisers that can give similarly high-quality input as [insert your intellectually favorite AI safety persons]. If models are to have this quality, they need to be good at conceptual reasoning. Again, this is true in general, but I think it makes extra sense from the “conceptual reasoning moves us from not obviously bad to actually good” worldview. 2: Training the part of safety that we