Capital Ownership Will Make You Go Crazy First

People worried about a “permanent underclass” often assume that owning capital will protect them from disempowerment, or at least that they will be disempowered last, after the jobless masses dependent on UBI. I think something else is more likely: owning large amounts of capital will get you disempowered sooner than average, just by a different path, namely going crazy.

How? The basic mechanism is actually easy to understand on a human level. One of the things difficult about being a billionaire is staying true to your values and keeping good epistemics. Why? Because a growing number of people are optimizing against your mind. Not “against” in an adversarial sense, but in the sense that your specific mind believing or doing something is part of their optimization objective. People see you as a source of resources and try to befriend you so you’ll fund them, invest in their startups, donate to their causes, or have sex with you. Nearby egregores extend their tentacles to convert you.

Staying sane gets harder. You often see countermeasures, like billionaires setting up family offices or small organizations that also act as filters, shielding them from part of the pressure (“I do philanthropy only through my foundation and don’t make personal donations, please don’t reach out”).

In my view most billionaires lose in this game, the question is just to what degree.

Now imagine a hypothetical future with the ultra-rich (or, as people usually imagine it, with you as the billionaire), where the rest of the world consists of the permanent underclass and countries of geniuses in datacenters. In this world, the content of your mind is the key to an absurdly large share of resources. Unless some magical social realignment happens, the countries of geniuses will optimize against your mind, and millions of desperate people with no better game in town will be happy to help them (or the other way around).

I think the default expectation is that you will go crazy. Not relative to your new values (you may feel great), but from the perspective of who you are now: what you currently believe and care about. Unless, of course, the only thing you care about is that some future you, with your name, keeps the power.The mechanism is similar to gradual disempowerment at the macro scale, just applied to individuals. What we believe and value is largely determined by culture, in a broad sense. Individually, we don’t have the capacity to observe most of the world, find abstractions, or get oriented on our own. We think in a collectively invented language, learn ethics socially, and even our interpretation of more biologically grounded signals, like emotions, is heavily culturally mediated.[1]

If humans somehow stay in control of all the wealth and political power, the content of our minds (what we believe, what we care about) becomes the key to the universe, the load-bearing part of the structure. With so much powerful optimization and abundant intelligence around, it seems likely that human-aligned culture will be replaced with something else. Political and economic power does not protect you against the content of your mind - if your mind goes crazy, power gets wielded by whatever beliefs and values took over.

As a post-AGI billionaire, can’t you protect your mind using AI? I think broadly yes, but roughly in proportion to how much your original human mind cedes control to your guardian AI. Again, the human analogues are easy to see: people wielding large amounts of nominal power, shielded by layers of guardians, protectors, and advisors. With AI, this means carefully filtering every bit of information, every pixel, sampling molecules from air you breathe and water you drink... You would likely also need to stop interacting with other humans, unless your protectors vet them extremely thoroughly and confirm they pose no risk of changing your mind. In the limit, what you believe and who you become are decided by AIs that are only nominally owned by you, the glorious meatpuppet.

Is there a better alternative? I think so, but not an individual one. Our survival and flourishing as individuals is tied to that of other humans, and largely depends on solving the civilizational alignment problem, including having a culture that is pro-human and pro-human-autonomy. Capital ownership alone won’t buy you a way to escape gradual disempowerment

^In fact, even our biology was partly shaped by culture, through selection similar to the domestication of animals.

In fact, even our biology was partly shaped by culture, through selection similar to the domestication of animals.