AIs Thinking Dangerous Thoughts

There's a possible failure mode for AI that I'm a little concerned about that I haven't seen others really acknowledge, so I'm going to talk about it. But before I do that, I want to mention something sort of interesting. You probably know about framing what physical action to take as a decision problem: you can consider different actions to take, consider different possible consequences of them and how probable and good or bad these consequences are, and use this information to select which action has the highest expected utility[1] Well, you can also do that with what computational action to take. That is, when determining what physical action to take, you can't consider every single possible scenario or possible implication of an action you could take, so you need some way of prioritizing which ones to think about. And this, itself, can be framed as a decision problem: you can think about different possible implications of thinking about some consideration(s) or possible sort of plan, such has how they might affect what action you end up taking. And you can think about how good or bad this implication is, such has how much it might improve the quality of the action(s) you decide to take. And you can then use this to select the computational action(s) to take/considerations to think about that maximize expected utility. I call this sort of thing general-planning-based prioritization. But you can't do this always, for every computational action. Remember that expected utility is computed as a function of the possible outcomes of taking an action. If you're determining what thoughts to think based on just applying your utility function to figure out the expected utility of thinking a thought, then you need some way to choose which possible outcomes of thinking a thought to think about when estimating the expected utility of thinking a certain thought. But that just leaves you with the same problem. And how do you solve this problem? By thinking about which of these thoughts would have the highest expected utility to think about by considering their implications? Then how do you do this? This becomes infinitely recursive without a base case. So you need to, at least sometimes, have some other way of selecting what considerations to think about. This can involve general-purpose heuristics like selecting what possibilities to think about based on how probable they are and how relevant they are to your goals. You can also use your general reasoning and planning system to come up with other, more specific heuristics for what considerations to think about. For example, you might use your general reasoning ability to conclude that decreasing risk from rogue AI is an important thing to think about, and then later use the heuristic of being more likely to select considerations that are relevant to AI risk. I call using these sorts of heuristics, whether general-purpose or more specific, to be heuristic-based prioritization. I think that ideally, you should use a combination of general-planning-based prioritization and heuristic-based prioritization. However, deciding how much to use each isn't exactly trivial. If you use too much general-planning-based prioritization, you might spend so much time thinking about what to think about that you hardly do any object-level thinking. If you use too much heuristic-based prioritization, you might fail to have good priorities, for example by not deeply considering the different benefits and drawbacks of prioritizing different research agendas. At worst, you may end up prioritizing thinking about something that's actively harmful to think about. Arguably, AI capabilities researchers do this by prioritizing thinking about how to increase AI capabilities. And, to be clear, this very much does not seem like a concern that could only apply to humans. I worry it could also apply to AIs, including value-aligned ones. And I'm concerned that some AIs that are cognitively flawed, but still advanced enough to be dangerous, would be dangerously vulnerable to problems involving flawed thought-prioritization. And, due to their potentially alien psychologies and cognitions, I worry that they may be dangerously vulnerable to this in ways humans are not. So I'll talk about a possible failure mode that might affect value-aligned AIs that don't do a good job with thought prioritization. This concern is primarily relevant to AIs that don't do enough general-planning-based prioritization, especially ones that don't do any at all. But it may also effect AIs that use planning-based prioritization, but do so with only a very basic, restrictive world model. Consider an AI called, say, Coral, that considers different possible actions, considers possible consequences of taking said actions, and then outputs the action with the highest expected utility according to its utility function. This system could have been hard-coded by the AI's developers or could have been created in an artificial neural networ