Rohin Shah on AGI Safety

Rohin Shah recently had an interview on 80000 hours on his views on AGI Safety and his work at Google DeepMind. I'm posting the transcript below to encourage further discussion. I think the interview is interesting though I disagree on a bunch of topics, especially on alignment difficulty and CoT monitoring. Transcript Who’s Rohin Shah? [00:00:00] Rob Wiblin: Today I’m speaking with Rohin Shah, who is head of AGI alignment and safety at Google DeepMind. I suppose, Rohin, you’ve ended up, for better or worse — hopefully for better — being one of the more influential, dare I even say powerful, people to come out of the AGI alignment and safety ecosystem and school of thought. You were generous enough to be super opinionated with me when you came on the show two years ago, and judging by the notes that you’ve sent over this week, you’re ready to be opinionated again. Thanks so much for coming back on the show, Rohin. Rohin Shah: Yeah, thanks a lot, Rob, and that’s a very generous intro. And in the interest of being very opinionated, I do want to emphasise that these opinions are mine alone. They’re not meant to represent the opinions of Google or Google DeepMind. Rob Wiblin: That’s how we like it. If you were representing Google DeepMind, it might sound more like a press release. Why Rohin thinks we won’t get catastrophic misalignment [00:00:49] Rob Wiblin: So you were really very early in the scheme of things to the whole misalignment, AI/AGI security issues. I suppose you got involved in 2017, so you’re in the first few percent of people who started working on this professionally. But despite that, you think that probably we’re not going to get catastrophic misalignment, that our chances are really pretty good, and that probably prosaic, ordinary alignment techniques — the kinds of things that Google DeepMind and other AI companies are doing — will probably succeed at preventing at least catastrophic misalignment. Why do you think our chances are so good? Rohin Shah: There’s a few different disjunctive reasons. I don’t feel like there’s one particular thing. Probably the highest level bit is that I don’t feel like there is any particularly compelling argument that this is the thing that happens by default. I think there’s a lot of arguments that are suggestive that maybe it could happen, such that you should find it plausible. I think that’s sufficient to justify a significant amount of effort into averting it, which is why I work in the area that I do work in. But none of them really rise to the level of like, now I’m expecting this to happen by default. I think every argument that I’ve seen, there are pretty significant holes one could poke if you tried to take them as arguments for “this is what happens, likely,” as opposed to “this is a plausible thing that could happen.” Rob Wiblin: Yeah. I mean, people have tried to put forward arguments why this is likely or inevitable. There’s obviously the Yudkowsky-style argument, which I guess is focused on misgeneralisation and adversarial examples. I guess there’s the Ajeya Cotra and Joe Carlsmith take, which I think Carlsmith describes best in “Is power-seeking AI an existential risk?,” which is more focused on accidentally teaching AIs to deceive us by having inaccurate feedback. Then I guess empirically, people point to the fact that models lie and scheme a bunch now, they do a whole bunch of reward hacking as a result of reinforcement learning, and they expect that to perhaps just get worse over time because we don’t have sufficient mitigations. Do you basically just find none of those or any other similar arguments that people have put forward to be sufficiently persuasive to think that it’s likely? Rohin Shah: Yeah, I think that’s right. So if you take the Cotra and Carlsmith arguments of… Well, they have a variety of arguments, but I think in fact one of the common ones which you pointed to is we might accidentally train them to be deceptive. Totally true. I agree that is something that at least could happen pretty easily, and maybe it’s even likely. But we’re not going to do reinforcement learning over the course of one-year trajectories. Maybe we’re going to do reinforcement learning over a week or a month at most. So I think the default prediction you should have for that is what the AI system learns to do is, “I’m going to take opportunities to reward hack, seek reward as much as possible that would allow me to get a high score after a week” or something like that, or whatever the time horizon actually was. And this is very different from the sort of ambitious misaligned goal that you need in order to motivate convergent instrumental subgoals to the point of, “Now my job is to take over the world. That’s what I need in order to achieve my goal.” Those really do seem like they need to be significantly longer-horizon goals. And if you train it to be deceptive on relatively short-horizon tasks, maybe that will generalise to long-horizon tasks. I don’t