Beyond Hardcoded Evolutionary Psychology

Steven Byrnes has written quite a lot on brain-like AGI algorithms. I'll reiterate here of a small part of his work, but you'd be better off reading his stuff directly. For the ideas which inspired me, see here, here, and here. This post has good handles for intuitively applying neuroscience to rationality. Don't mistake my "simulator" cluster for a distilled and reduced concept; it's "lumpy rock", not "skewed prism of seafloor impactite from the late Devonian". My (loose) extension of Steve's models of human minds predicts that we'll get stuck in local minima of self-reinforcing nonreality; with this knowledge, we can more directly target the underlying mechanisms of cognitive biases. Consciously tracking what you're actually optimizing for is one example skill which seems worth developing despite its time cost. In fact, many rationality skills (like noticing your confusion and scout mindset) are species of this overlying stance. Let's take scout mindset as an example. While discussing something, the goal for rationalists usually isn't to convince the other person of their ideas, but rather to come to truthier beliefs. One should therefore be vigilant, lest they fall into a combative posture, since this makes them slow to update. Cool! Is there a principled way to train this? Do you mean trigger-action patterns? No; TAPs take conscious effort to implement. How do you make it an unconscious System 1 response, the same way I keep my balance while walking? Lots of practice. But what if I have stronger subconscious pressures pushing me away from scout mindset? Like, sometimes I want some idea to work, and then I notice myself defending it. But then it feels like something blocks me from seeing that defensive posture. It's like a part of my brain is trying to keep me blissfully unaware. You're telling me to run uphill, but I want to remove the hill entirely. If you keep practicing it, you'll learn to do it subconsciously! Then it won't take any effort, just like walking. But as an infant, I wanted to walk; it got rewarded. Rationality isn't emotionally satisfying on the timescales I care about, at least not by default. My conscious practice builds a sandcastle, then tomorrow the tide washes away all but a shallow lump. Sure, after 3 decades my hill might rise above the waves, but what stops me from building a levee?[1] Perhaps modern neuroscience can help this person be rewarded by rational thoughts? Let me just... Simulators Brains run simulations. I've not seen Steve claim this[2], hence "simulations" instead of "thought predictors". Let's say that I go parasailing in the Andes. During flight, my mind absorbs tons of data about the sky, landscape, my gear, etc. When I get home, my brain runs lots of internal simulations about flying through the mountains, birds, and more specific stuff about, say, wind patterns. These simulations are based off of those earlier sensory experiences. In this way, my mind converts the tidbits of data in my flight memories into lots of training examples for itself. To a Solomonoff inductor (an imagined perfect intelligence), my simulations don't add any extra information about my flight compared to my memories. But I'm not a Solomonoff inductor, so my mind needs many self-generated training examples to fully internalize it. (This is one reason we're so damn sample-efficient compared to LLMs. We can keep generating internal training data. LLMs, by comparison, have to be fed training data by some externally bottlenecked process like internet scraping or human-written RL environments.) From what I see, LessWrong rationality advice is almost entirely "ingest this particular training example", which works great if your simulators are faithful to the content of that data. But they're not. Simulator dynamics cause cognitive biases and actively prevent you from fixing them. Drift; cognitive attractors Simulators can be influenced by each other cyclicly; they're not straightforwardly truthful updates on sensory data. As an extreme example, these internal thoughts[3]: Blah unfocused daydreaming blah If my best friend of 15 years attacked me with a frying pan, I'd probably punch him in the face and try to dodge the pan since it's slow to swing a cast iron pan like that Imagining friend misses and shatters a window with the pot momentum "Bro, stop!" Now I like my friend less! He did absolutely nothing to imply that he'd attack me with a frying pan. In fact, I had a stomache cramp I didn't notice, which biased my idle thoughts towards pain, and my friend happened to be on my mind because I visited him that morning. A more typical example for the audience of this post: After a conversation with a non-rationalist family member about my decision to drop out of my undergrad math degree She just doesn't understand. AI safety is too weird of a subject, and she's stuck on immigration or whatever Facebook feeds her all day. Option value What's that? Oh, I lost my train of thought. Why can'