TeX was invented to typeset math but is now used for reasoning

I'll keep this short, since it's a simple observation that I haven't seen anybody else make, about the way computer systems do math.

If you ask a language model to do a multi-step math problem (let's take GLM-5.3 as an example because you can see the entire CoT — nothing up its sleeve), you might see something like this:

We need to evaluate the integral $\int_0^\infty \frac{x^3}{e^x - 1} dx$.

The standard approach: Use the geometric series expansion. We have $\frac{1}{e^x - 1} = \frac{e^{-x}}{1 - e^{-x}} = \sum_{n=1}^{\infty} e^{-nx}$ for $x > 0$.

What are all these symbols like \int, \infty, \frac...? They're TeX of course!

Donald Knuth created TeX to typeset math, you know, for display. It had nothing to do with the actual computations, which would either be done with pencil and paper,[1] or else with Mathematica or Maple or something, which work completely differently. If you'd asked Knuth in the 1980s about doing algebra in TeX he'd have looked at you very funny because the idea doesn't make sense.[2]

Then, generative language models were trained on corpora including many TeX/LaTeX documents and learned how TeX works (and more importantly the mathematical meaning of the symbols).

So, nowadays when an LLM solves a math problem, it will very often use TeX for the intermediate steps of the math problem, like it's actually manipulating the TeX symbols in order to reason. We can tell it's using it this way because it happens in the CoT, which for many AI products is never rendered/displayed.

So we ended up in a world where the most powerful mathematical problem solving agents (e.g. the ones behind the recently claimed Navier-Stokes solution) reason using TeX.[3]

(Knuth has written briefly about generative AI, but I don't think he's ever commented on this unexpected development... and I'm not sure if he's even aware that Claude's solution to his own problem was very plausibly produced by Claude manipulating and "thinking in" TeX.)

^Of course, pencil-and-paper computation uses the same notation that TeX encodes, since TeX didn't invent ∫, or "sqrt" for √. But the point is that TeX became the dominant plain-text, 1D encoding of this traditional 2D math notation, and so LLMs learned to use it... for thinking, not just typesetting.

Of course, pencil-and-paper computation uses the same notation that TeX encodes, since TeX didn't invent ∫, or "sqrt" for √. But the point is that TeX became the dominant plain-text, 1D encoding of this traditional 2D math notation, and so LLMs learned to use it... for thinking, not just typesetting.

^Well, he might actually have said "oh sure, you mean like macros and stuff? like how TeX is Turing-complete?" and then when you clarified that you meant a non-deterministic system that doesn't even run a TeX engine at all but still directly manipulates the TeX symbols in order to do algebra... then he would have looked at you really funny.

Well, he might actually have said "oh sure, you mean like macros and stuff? like how TeX is Turing-complete?" and then when you clarified that you meant a non-deterministic system that doesn't even run a TeX engine at all but still directly manipulates the TeX symbols in order to do algebra... then he would have looked at you really funny.

^They also published a formal Lean proof, so you might say that the strongest agents reason in TeX but verify in Lean. "TeX for thinking, Lean for checking."

They also published a formal Lean proof, so you might say that the strongest agents reason in TeX but verify in Lean. "TeX for thinking, Lean for checking."