AI risk and the rational voter

A common reaction to arguments about AI risk is disbelief that anyone would let it happen. If advanced AI really threatened everyone, surely people would recognize the danger and act to prevent it. Would they really sleepwalk into catastrophe when avoiding it is in everyone's interest? I think voter behavior in democratic countries is a good analog. It is no secret that voters are remarkably ignorant about policies, politicians, and basic political processes. Voting decisions are heavily influenced by candidate charisma, height, inspirational speeches, attack ads, vibes, and mood affiliation. Naively, this behavior does not seem very rational. But this is using the wrong notion of rationality. Most votes have very little impact on the outcome: a single vote has an extremely small chance of deciding a race, and most seats and races are safe anyway. So it is not usually rational to vote for the purpose of changing political outcomes. What would be rational [1] is to vote in ways that make the voter feel good about themselves. This is sometimes called expressive voting. Bryan Caplan's The Myth of the Rational Voter pushes this logic one step further, from votes to beliefs. It takes a lot of work to be well-informed about public policy; some people devote entire careers to specific aspects of (say) monetary policy or international relations. Since the average voter's impact on the election is very small, it really doesn't make sense for them to invest any time in understanding the issues at stake just so they can make a marginally better decision. It's much better to simply not learn anything at all and use the beliefs you already have. And since false beliefs are not costly, voters are free to indulge whatever feels most intuitive or makes them feel best. Caplan calls this rational irrationality. [2] Voting is an extremely crisp illustration of this behavior, but it appears in many other contexts. People's opinions of historical figures are often a bit of a dumpster fire; consider the Che Guevara t-shirts, or the Stalin tankies. Supporting a long-dead figure is even more inconsequential than voting. [3] We are not resurrecting Stalin any time soon, so if he makes you feel warm and fuzzy for whatever weird reasons, who the hell cares? Likewise, false beliefs about climate change, astrology, or even vaccines [4] are actually not that costly to individuals. For most people, who have no particular material stake in AI, the situation looks much the same. It is not obviously rational for any individual person to have correct views on AI risk: it is very unlikely that the actions they could take would change the outcome. Whether they in fact hold correct beliefs will instead depend much more on whether those beliefs fit with the internal psychological and external social forces at play. Does the belief make them feel good directly, or through justifying some narrative about themselves? Will it make them seem clever and contrarian, or, alternatively, weird and unhinged? The rational response, though, is probably just to ignore the issue entirely. Then there are people with more specific, though still weak, reasons to dismiss AI risk. Since the effect of their beliefs on the actual risk is again negligible, weak reasons suffice. For some, AI risk is threatening to prior ideological preconceptions, and much like voters rejecting policies that are ideologically alien, they happily reject the risk. The typical Alphabet or Nvidia shareholder likewise receives a small but tangible benefit from their shareholdings, in this case monetary rather than psychological, and this is more than sufficient to overwhelm any potential risks to themselves. Lab workers have a substantial stake in AI. In this way, they are much more analogous to a party operative or backbencher than to a voter. For them, it can still be fairly rational to ignore extinction risk, because the typical lab worker really is only having a small effect. Even if the average lab worker were increasing extinction risk by 0.001% by accelerating capabilities, it seems quite plausibly rational to round that to zero in the face of the overwhelming incentives otherwise. Perhaps a few key actors—lab leaders, senior political figures, and some specific employees—could make decisions with real stakes, such that the costs to themselves are non-trivial but the benefits substantially greater. If they are sufficiently reckless, they might just rationally be willing to take the risks for their own selfish gain. Are we doomed? We don't have to be. Humans have built communities and institutions that reward selfless behavior many times before, with varying degrees of success. With the right structures, even ordinary, selfish individuals may be motivated to act in the collective interest, and people whose interests genuinely include the welfare of others can be empowered. But such structures do not arise automatically. Individual rational s