Pretraining data, not verifiability, is why LLMs are especially good at math (and coding)

A common take I’ve been hearing is: “LLMs are especially good at math[1] because math is easy to verify”. But that story doesn’t make much sense.

For one thing, “easy to verify” only matters for the RL part of LLM training pipelines, and the leading LLM companies have said that they spend very little effort on RL-for-math.

Worse, to the extent that the companies are doing RL-for-math, it’s RLAIF, not RLVR.[2] So really, the phrase “math is easy to verify” amounts to “LLMs are very good at judging math arguments”. But that’s begging the question! Why are pretrained LLMs so much better at judging math arguments than judging, say, fiction writing? We still need an answer.

LLMs are especially good at math because almost everything in the math literature is correct. Read a random sentence in a random math paper in the research math literature, and you can be >99% confident that the sentence is true. So if LLMs do what they do best—imitative learning—then in the limit, they’re going to be almost always making correct deductions in their mathematical reasoning. They just need a little sprinkling of curated mid-training data and/or RL to hone their metacognitive strategies and so on.

So that’s my theory: it’s about pretraining data, not verifiability.[3] Now let’s apply that theory to other areas.

What about code? Well, for almost all the code that you’ll find on the internet, it compiles and runs and basically does what you’d expect it to do. So if LLMs do what they do best—imitative learning—then in the limit, they’re going to be almost always writing code that compiles and runs and basically does what you’d expect it to do. But code on the internet is also often ugly, buggy, inefficient, etc. So companies need to solve that problem with curated mid-training data and/or RL. This is a bigger challenge for code than it is for math, but companies have spent the past few years pouring a ton of effort into it, and they’ve made impressive progress.

What about other fields? In many fields, the research literature is kinda a dumpster fire. There’s plenty of true and valuable stuff, stirred into a sea of such falsehoods and confused ideas that reading them will make you stupider. So if LLMs do what they do best—imitative learning—then in the limit, they’re going to spit out tons of confused nonsense with occasional insights, with no labels on which is which. Just like the humans. You might reply: “Perhaps this problem is fixable via curated mid-training data and/or RL?” Maybe. But if so, I think we should expect it to be much more of a challenge than it has been for code, much less math.

How does this theory bear on the “LLM Recursive Self-Improvement” debate?

If you think that the path to superintelligence involves coming up with lots of mundane incremental ML ideas, then the question to ask yourself would be: Is the “mundane incremental ML ideas” research literature more like the research math literature (pristine fountain of truth), or more like code on GitHub (everything basically works even if it has issues), or more like the research literature in many other fields (dumpster fires)?

If you think that the path to superintelligence involves coming up with radically new AI paradigms, then the question to ask yourself would be: Is the “radical new AI paradigms” research literature more like the research math literature (pristine fountain of truth), or more like code on GitHub (everything basically works even if it has issues), or more like the research literature in many other fields (dumpster fires)?

I shall leave readers to reach their own conclusions.

How does this theory bear on the “LLM Automated Alignment Research” debate?

Same idea here. Again, readers may judge for themselves.

^I mean, compared to other things that LLMs do. I’m not comparing LLMs to human mathematicians in this post; I have opinions but that’s a separate debate.

I mean, compared to other things that LLMs do. I’m not comparing LLMs to human mathematicians in this post; I have opinions but that’s a separate debate.

^“RLAIF” (RL from AI Feedback) is an LLM judging the output of an LLM. “RLVR” (RL with Verifiable Rewards) would involve some mechanical checker, like the Lean formal verifier, in the RL loop. I don’t have insider knowledge, but my strong impression is that most-or-all of RL-for-math training involves RLAIF, not RLVR. E.g. see a discussion by Google here.

“RLAIF” (RL from AI Feedback) is an LLM judging the output of an LLM. “RLVR” (RL with Verifiable Rewards) would involve some mechanical checker, like the Lean formal verifier, in the RL loop. I don’t have insider knowledge, but my strong impression is that most-or-all of RL-for-math training involves RLAIF, not RLVR. E.g. see a discussion by Google here.

^Of course, “verifiability” is relevant to how humans wound up with such a high-quality math literature. So verifiability-of-math is indirectly related to LLM math competence. Just not directly.

Of course, “verifiability” is relevant to how humans wound up with such a high-quality math literature. So verifiability-of-math is indirectly related to LLM math competence. Just not directly.