Extreme Rationality: Still Not That Great

The tl;dr has spoilers, so I've put it at the end.

Also feel free to skip any of the chapters because the post turned out to be very long. I think you can read almost any of them on their own, and you can skip to On practicality if you want the big picture.

The title is stolen from Scott Alexander's post Extreme Rationality: It's Not That Great. Basically, it will be my opinion on applied rationality and its successes and failures. The title speaks for itself: I think there are few successes. A few weeks before Scott's post, Eliezer wrote A Sense That More Is Possible where he urged developing training for rationality; and though he thought rationalists were just ordinary people back then, he believed they could become much more. Center for Applied Rationality (CFAR) was created in 2012 with the goal of developing such an Art. So I will discuss rationality's basic values, where I disagree with them, and where they led.

But first, let me start with my personal story.

I was one of the main organizers of LessWrong events in Saint Petersburg, Russia, for two years. I started by reading HPMOR and the Sequences and running discussion groups about them. But then I took a CFAR-like workshop at Kocherga (Moscow) and became obsessed with applied rationality: I tried to practice Hammertime and CFAR handbook. After some time I decided I had enough knowledge to teach rationality, and together with a friend I developed an educational program, which failed for multiple reasons. I also studied bioinformatics and worked for two years at Gero, an anti-aging research company, precisely because I had read HPMOR and been inspired by transhumanism (I still am).

Time passed; I was in psychotherapy and realized that a lot of the things I had been trying to achieve were very unhealthy. I started to criticize applied rationality for that—partly because it is quite comforting to blame a group's ideology instead of your own personal problems. Since I was aware of that bias, I was curious to actually figure out how damaging CFAR practice had been to me and to other people. But I didn't focus on it too much: I just wrote a couple of small posts a few years ago and that was it (they were in Russian, and basically mirrored the posts of another co-organizer of rationality courses in Moscow). Recently I recorded a podcast (also in Russian) with the ex-leader of the Moscow LessWrong community Slava Matyukhin, who supported a lot of my claims, and I became really curious about what is going on—and what had been happening—in the English-speaking part of the community. So I started to assemble the complete picture.

I will try to understand: are people really so biased and irrational?

I will talk about what values the rationalist community implies, and what I think about them.

I will discuss the practicality of CFAR: did it help people in the end?

Finally, I will talk about the promises and the sense of self-importance on LessWrong, and draw some conclusions.

There are a few posts that I think intersect with what I am saying here, but I am not going to cite them: Rationalist Epistemics and the Sequences (Effective Altruism Definitions Sequence), Rationalist Epistemics and Social Epistemology (Effective Altruist Definitions Sequence), The Rationality Wars.

The first and foremost bias, the cornerstone of Kahneman and Tversky's theory, is Loss Aversion. When I only started to develop my understanding of the topic I stumbled upon a post by Jared Peterson: Biases Don't Exist, and Humans Are Not Irrational. It was debated there whether biases are actually bad things, because a bias is just a deviation from utility theory, which is not always right.

One example for a failure of utility theory he proposed is ergodicity. You probably know the game where you flip a coin: heads gives you a 50% increase to your bankroll, and tails decreases it by 40%. This process is non-ergodic, and the expected value of each flip is positive (+5%), yet almost every individual trajectory decays to zero, meaning you definitely lose everything. Any trader understands that you can't just maximize the utility of the next bet; some risks are not worth the cost, because after a couple of tries your account is depleted and you never recover.

A simple explanation of the coin example is that, with large N, you get the following (a 50% probability of adding 50% means utility is multiplied by 1.5, and a 50% probability of losing 40% means utility is multiplied by 0.6):

So utility decays with N at a rate of the square root of 0.9.

Another related caveat is that humans actually have diminishing returns on everything, so it is fine to fear losing a given amount of money more than you value not gaining the same amount.

But as it turns out, all of this is not new: Bernoulli invented diminishing returns while trying to solve the St. Petersburg paradox in 1738 (interestingly, the city of St. Petersburg was only 35 years old back then). The St. Petersburg paradox is a game of coin flips with a casino: it pays you dollars each time tails appears, where n is the flip number, and the game ends when heads appears. The expected payoff of this game is infinite (I'll just assume you can handle the series in your head). But for some strange reason, people rarely pay more than $25 to enter such a game. The solution is the invention of diminishing returns, and it can be any concave utility function: the more you have, the less additional utility you get from the same amount of money.

Ergodicity was introduced by Ludwig Boltzmann in 1871. Though I was too lazy to research when it was introduced into economics, I found a fairly recent paper that summarizes its effects well: Peters, O. (2019), "The ergodicity problem in economics." Nature Physics. The conclusion of interest for us here is that ergodicity forces us to use a log utility function, which is equivalent to accounting for diminishing returns — an adjustment already widely used in economics.

So did Kahneman and Tversky account for that as well? If you look at their original paper — Tversky & Kahneman 1992, "Advances in Prospect Theory" — they model utility in a fairly complex form: for x ≥ 0 for x < 0

Where x ≥ 0 is the utility of gains and x < 0 is the utility of losses. I am not sure if it is very important that the logarithm function required by ergodicity doesn't fit well here, but it definitely accounts for diminishing returns, and ergodicity gives a quite similar effect anyway.

And even though we kind of accounted for these two things, we still see two weird effects here.

First is the reflection effect: the fact that the function is convex on the negative side instead of concave, and the tricky part is that this change is relative to your current state, or reference point. On the standard theory you would not see anything special at the reference point at all: your utility function starts at zero wealth, and from there you have just log utility. It would not depend on whether you lose or gain in a particular scenario, but on the level of wealth it leaves you at. But humans fear additional losses less and less, regardless of how much money they start with. Also, the function should not have become convex. More plainly, you can describe it as risk-averse behaviour for gains and risk-seeking behaviour for losses. People want to risk more even when expected utility is negative, in cases like the choice between (−$4,000, .80) and (−$3,000 sure). Because the jump from −3000 to −4000 is less scary than the jump from −2000 to −3000, they may choose −4000 even though 4000 × 0.8 = 3200. There is actually another effect at play here, probability weighting, but I don't want to go into it.

Second is the kink. It is the fact that we have a lambda coefficient in front of losses, which tells us how much more losses loom than gains. It is not as straightforward as saying that losing a given amount of money has lambda times the weight o