LLMs are (still) mostly powered by imitative learning, not RL

Reinforcement learning from verifiable rewards (RLVR) is the hot new thing in LLM training. It's so hot, and people spend so much time talking about it, that they sometimes lose sight of the big picture.

Stepping back, LLMs can do lots of very impressive things. How? Where did those capabilities come from? Fundamentally, they come from a combination of:

(1) Imitative learning, including pretraining and supervised fine-tuning (SFT). See my earlier discussion: "LLM pretraining magically transmutes observations into behavior, in a way that is profoundly disanalogous to how brains work".

See my earlier discussion: "LLM pretraining magically transmutes observations into behavior, in a way that is profoundly disanalogous to how brains work".

(2) Reinforcement learning, including RL from human feedback [RLHF], RL from AI feedback [RLAIF], and especially RLVR.[1]

If we look at the final trained LLM, we can ask how important each of those two pieces was, in explaining the LLM's capabilities. And my claim is that it's way more (1) than (2).

I'll start in §1 with some relevant evidence, and then in §2 I'll circle back to operationalizing exactly what I'm claiming, and finally in §3, three reasons why we should care—namely, it affects how we should think about chain-of-thought legibility, about LLM capabilities, and about LLM alignment.

Note that I am not arguing that RLVR does not importantly contribute to LLM capabilities. That would be absurd! Of course it does! Companies use RLVR because it works, and I expect them to continue doing so more and more. Again, the things I'm actually claiming are in §2–§3.

1. Some relevant evidence

1.1 Theoretically, each GPU-hour spent on RL should have orders of magnitude less contribution to LLM capabilities than a GPU-hour spent on imitative learning

The fraction of total LLM training compute that goes towards RLVR is high (I vaguely recall hearing >50%). But that's different from how to divvy up the credit that pretraining/SFT versus RLVR can take for LLM capabilities. Why is that different? See:

Based on those posts, if RLVR is 50% of the training compute, then it might only be 0.01% of the information content imparted by training. And unless the learning rates are drastically different, it would also be 0.01% of changes to the weights.

1.2 The chain-of-thought (CoT) is still obviously strongly influenced by imitative learning

People emphasize how frontier model CoTs can be pretty weird, and that's true and important, but if you step back a bit, in the grand scheme of things, CoTs are still very obviously strongly influenced by imitative learning from human-created (and hence legible) text.

I prefer to start from the opposite side. What would a really inscrutable CoT look like? It would look like random tokens: "... extracurricular(boolean ger застос autism relevance directamenteдвэр QUICKchem ..."

CoTs sure don't look like that! So on a spectrum from "totally legible" to "looks like random tokens", we are way closer to the "totally legible" side, as of today.

What does that imply about how LLMs get their superpowers? Well, imitative learning by itself will necessarily create legible CoTs; whereas in RL, the CoT is generally irrelevant to the reward, so there's no force keeping it legible, and thus we should expect it to drift into an ever less legible and more "optimized"[2] direction.

Thus, the fact that CoTs are still pretty legible is some evidence that RLVR today has not had that much influence on LLMs, as compared to imitative learning.

1.3 LLM companies still seem to care a lot about imitative learning (pretraining & SFT) data, not just RL environments

See "Most Algorithmic Progress is Data Progress" (Beren Millidge 2025), plus further discussion in my post "The nature of LLM algorithmic progress" §1.4.

And LLM companies today continue to spend money on even more training data for imitative learning (especially expert reasoning traces for SFT, I think).[3]

1.4 Three papers claiming that non-RLVR'd models can get into the same ballpark of capabilities as RLVR'd models, although maybe we shouldn't trust those papers too much

"Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?" (Yue et al. 2025; site, arxiv, lesswrong discussion):

The authors found that when an RLVR'd model could solve a problem, the equivalent non-RLVR'd model could generally solve the same problem, maybe not on the first try, but in one of the first 1000 tries or whatever.

The authors found that an "iterative sampling algorithm" could boost a non-RLVR'd model to be comparable to the corresponding RLVR'd model.

The authors found that you can take a non-RLVR'd model, randomly perturb the weights, and ensemble the results by majority vote, and you can get results comparable to an RLVR'd model.

Caveat for all of these: I don't think we should over-anchor on any of these, mainly because they're all now outdated, and based on open models rather than bleeding-edge LLMs with bleeding-edge RLVR best practices.

Also, there's another paper "ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models" (Liu et al. 2025, arxiv) which claims "that RL-trained models consistently outperform base models … including scenarios where base models fail entirely regardless of the number of attempts".[4]

1.5 A paper suggesting that RLVR mostly refines the heuristics controlling which (already-known) reasoning strategy to use in which situation

"Base Models Know How to Reason, Thinking Models Learn When" (Venhoff et al. 2025; arxiv)

Using interpretability tools, the authors conclude "RL primarily teaches heuristics for orchestrating pre-existing base mechanisms, whereas SFT-distillation installs new ones".

I think the basic idea is that, to do well in long-chain-of-thought reasoning, there are a bunch of moves that you can make: press on, backtrack, reexamine assumptions, pivot to Alternative Strategy X, etc. The LLM "knows" all those moves from imitative learning, but doesn't have refined heuristics on which move to make in which situation. These kinds of heuristics are very important for making progress in a long CoT, and they're somewhat domain-specific, and you can't really pick them up well by imitative learning. So that's the main low-hanging fruit that RLVR fixes, and it makes a huge difference in practice.

Caveat: Just as in the previous section, this paper is old (by RLVR standards), and not based on bleeding-edge frontier LLMs with bleeding-edge RLVR best practices. I guess in this case I'm slightly reassured by the fact that one of the coauthors, @Neel Nanda, works with near-SOTA LLMs at his day job at DeepMind. But only slightly.

Anyway, I'm taking this claim as a good default mental picture, not an ironclad eternal truth. In particular, I remain open-minded to the Liu et al. 2025 claim that "RL can indeed discover genuinely new solution pathways entirely absent in base models, when given sufficient training time and applied to novel reasoning tasks." (But I'm also open-minded to the opposite! Liu et al. make that claim but I don't think they really prove it.)

(For more discussion and references on this topic, see "Some Interesting Papers on RLVR" by @CarolusRenniusVitellius.)

2. What am I actually claiming here?

What's my point? I feel most strongly about the following intuitive picture:

We start with a randomly-initialized LLM. It "knows" nothing at all, and regardless of context, it will outpu