How much will conceptual reasoning training transfer to ML research abilities?

Recently the Conceptual Reasoning Capabilities (CRC) team at Redwood released a post arguing that we should accelerate AI’s conceptual reasoning abilities. This post was very controversial. In particular, people are worried that improving AI’s conceptual reasoning abilities will accelerate both their ability to do ML research and the capabilities they need for takeover.

I am going to address just one chunk of this controversy: how much does training conceptual reasoning on AI safety questions translate to general ML research abilities? As a reference point for conceptual reasoning training, we can use the example of LMCA questions. These questions measure conceptual argumentation by having models rate counterarguments made against a claim. Counterarguments are rated on axes like strength, clarity, and centrality. The claims and counterarguments are in a variety of conceptual domains, including AI safety and philosophy. Using LMCA, I can provide a more specific operationalization of the question I’ll be tackling: if you train on LMCA-style tasks where all the arguments are AI safety relevant until you get a factor of X increase on automated AI safety work, what factor increase do you get on automated ML research?

I am going to tackle this question by breaking it down into what I consider to be the primary considerations and quantifying the contribution of each. I am hoping this approach will make the discourse easier to navigate and help readers identify the most important cruxes. So far, I have found this controversy hard to pin down: good points are being made on both sides, but I don’t know how much weight anyone gives to any particular point. Thus, my numbers are meant to make it easier for people to see exactly where I stand on each subpoint (even though I have large amounts of uncertainty).

Conceptual reasoning skills are partly domain-specific. Training conceptual reasoning on AI safety questions increases conceptual reasoning on other domains less heavily.

ML research is less conceptual than AI safety research. Training conceptual reasoning on AI safety questions might transfer less to ML capabilities because ML is a fundamentally less conceptual skill.

ML research will be more saturated than AI safety research. The kind of ML research skills that would be improved might be redundant with those already learned via RLVR loops (more so than for AI safety research).

We can measure domain shift by training Opus 4.5 on LMCA-style tasks until it gets a 20-point increase on its LMCA score (bringing it to ~Fable 5.1 level), but for both training and testing we only use questions in the AI safety domain. Now we test this model to see how much its LMCA score improves for questions in a similarly conceptual domain, for example, (arguably) economics.

To account for AI safety being a more conceptual field, as well as the amount of domain shift, we can imagine training Opus 4.5 on LMCA-style tasks within the domain of AI safety and then measure how much its increase in general AI safety skills (not only conceptual AI safety) compares to its increase in general ML research skills.

We can factor in saturation effects by imagining that Opus 4.5 goes through a significant amount of post-training on ML research and adjusting our answer based on this effect.

These operationalizations are not perfect. For example, they do not account for how effects will change as models become larger and smarter. We also have to imagine that our measurement for AI safety capabilities and our measurement for ML research capabilities have equivalent translation to real-world speedup.

It’s also possible these three considerations miss something important. I chose them based on topics that came up in the CRC team’s post, the responses to it, and my conversations with the CRC team.

One way we might be able to reduce improvements to ML research abilities is simply to avoid training on conceptual questions directly related to ML. The CRC team notes that they have updated downward on the amount of generalization that is likely to happen between domains:

We don’t necessarily expect much generalisation between different conceptual domains. For instance, some AI alignment work seems relatively far from moral philosophy. And empirically, our sense is that generalisation from small-scale training is pretty weak. Our view on this has evolved over time: based on empirical evidence, we have gradually updated to expect less generalisation.

One way to understand why conceptual reasoning may not generalize across domains is that a better understanding of the domain may be just as helpful (if not more so) than general reasoning skills in helping an AI score higher on LMCA-style questions. For example, one of the provided LMCA questions discusses the is-ought gap, and in order to answer it well, you have to know the difference between descriptive and normative claims and also be able to identify exactly which changes of phrase convert a descriptive claim into a normative claim. It’s possible that LMCA-style training largely teaches domain-specific skills that won’t transfer much to other domains.

If conceptual reasoning training works largely via pretraining elicitation, then there might be a larger amount of generalization. We can analogize this to emergent misalignment, where training on insecure code brings out a malicious persona that has wide-reaching effects. In this case, traits like “careful thinking” that are helpful in a wide variety of domains might be selected.

The best empirical analog I know of finds a large amount of generalization. In Does Math Reasoning Improve General LLM Capabilities?, they trained Qwen3-14B-Base on math only and got a 0.8 ratio between the improved score on math evaluations and the improved score in other reasoning domains (including logical reasoning, planning, coding, and science questions). However, there are reasons to think this number might be higher than the LMCA transfer to ML. The main reason is that the harder the optimization for a particular skill, the less that optimization generalizes, as overfitting begins to set in. And I expect less generalization from the CRC team adding training on top of a frontier model than the authors got for a Qwen3 base model, both because the Qwen3 base model was less aggressively optimized in general and because I expect the CRC team to apply more optimization pressure than occurred in this study.

Weighing these considerations, I think they justify a small downward adjustment to the empirical anchor of 0.8. Thus, I expect about half as much improvement out of domain as in domain. Concretely, I predict that training Opus 4.5 on AI safety LMCA questions until its LMCA score on AI safety questions rises by 20 points would raise its score on LMCA questions in similarly conceptual domains by about 10 points.

The CRC team argues that ML research is much less conceptual than AI safety research because you can get verifiable feedback for it, and when you can't, the insights you need are still more heavily informed by empirical data than a priori reasoning. As evidence, they note that AI companies seem to mainly hire ML researchers on the basis of empirical skills rather than conceptual reasoning ability.

Before discussing this point further, I want to clarify that I am disentangling “ML research has more verifiable signals, and so it will be saturated by RLVR” from “ML has more verifiable signals, and this is indicative of it being a fundamentally different kind of domain, and one where conceptual reasoning skills are less relevant." In this section I am only discussing the latter.

It’s not clear that the presence or absence of verifiable feedback is load-bearing for the kind of skills that are at play. It’s possible that knowing how to weigh different kinds of empirical data and then extrapolating from there is very similar to weighing different kinds of arguments. The CRC team begins by defining conceptual reasoning such that it excludes domains with empirical feedback and then argues that ML has high reliance on empirical feedback, but they never argue that the way they defined conceptual reasoning captures the relevant properties for predicting how capabilities will generalize. To be fair, this distinction becomes relevant in regard to estimating the amount of saturation (which I address in the next section) and this is likely why they emphasize it. Nonetheless, it’s worth noting that reasoning about data and reasoning about arguments could be quite overlapping.

While AI companies may primarily hire ML researchers on the basis of empirical skill, this also seems to be true for their AI safety researchers. In Talent Needs of Technical AI Safety Teams, Ryan Kidd notes that both for safety teams at AI companies and for independent researchers, good empirical iterators are valued particularly highly. If we break down a team's skill needs into the time that researchers spend on different kinds of activities, AI safety research and ML research also look similar: Fable 5.1 estimates that about 25% of frontier labs ML researchers' time is spent on conceptual activities compared to 35% for AI safety researchers (these activities include choosing research directions, designing experiments, paradigm-level thinking, and writing).

In light of these considerations, I don’t expect a large drop in generalization of conceptual reasoning training to ML on account of ML being less conceptual (at least relative to AI safety). I estimate a factor of .75 on account of this consideration (roughly matching Fable’s estimate of the ratio in time spent on conceptual work). Combining this factor with that from the last section, I predict that training Opus 4.5 on LMCA questions until its general AI safety capabilities rise by a factor of X would raise its general ML research capabilities by factor of .5*.75=.375X.

The CRC team’s stance is that even if conceptual reasoning tasks could generalize to improving ML capabilities prior to ML-specific training, AI companies will have already been able to reap the large majority of these improvements from verifiable ML tasks. This is both because a particularly large amount of resources will be put toward improving ML capabilities and also because ML is particularly verifiable so the standard RLVR signals may more comprehensively improve the relevant skills.

On the other hand, the CRC team does mention a couple kinds of less verifiable reasoning abilities that do seem relevant for ML capabilities. One is answering highly conceptual ML questions like “whether next-token prediction can in principle scale to ASI." Another is “structuring very large quantities of knowledge (say, thousands of papers) and organizing very large research fields.” However, the CRC team argues that these conceptual contributions are more relevant for figuring out what works in principle than in practice, and that these types of in principle considerations are less likely to bottleneck ML research capabilities than the types of in practice considerations that can largely be informed by small scale experiments with verifiable endpoints.

Even though we have verifiable signals for ML research (such as running small scale ML experiments and rewarding improvements in sample efficiency and compute efficiency), those signals may have disadvantages like expense and bias such that there is still alpha in using LMCA-style training. Expense is pretty straightforward: having an AI iterate on training its own models would be costly, particularly if those training runs have to be large in order to provide useful insights. Conceptual reasoning training might provide a warm start that saves a large amount of expense by providing more signal per dollar in early training. The signal for RLVR training might also be significantly biased by what the model chooses to explore. Conceptual reasoning training could provide useful priors for exploration and introduce novel skills that would have been missed in an RLVR-only setup.

There are fundamental problems in ML (e.g., low sample efficiency relative to humans) that may require something like a paradigm shift to solve. Some conceptual skills, like reasoning out of distribution, seem useful for making these leaps. At the same time, models may be less likely to learn the conceptual taste needed for a paradigm shift under RLVR, because effective paradigm shifts are hard to find through local exploration. Thus, conceptual reasoning training might be a good source of signal for these skills. This is concerning because tackling fundamental ML problems effectively might be especially acceleratory.

In spite of my pushback, I still think this is one of the strongest considerations and I estimate it would decrease effective learning transfer to ML by a factor of 4. Thus, my assessment, combining the three factors I have considered so far is that training a heavily ML-research-post-trained version of Opus 4.5 on LMCA questions until its general AI safety capabilities rise by a factor of X would raise its general ML research capabilities by a factor of .5*.75*.25≈0.1X.

The numbers I give here are a long way from answering the question “Is the CRC team’s work good?” Other relevant considerations include, the amount of speedup conceptual reasoning provides for other useful domains such as philosophy and decision theory and the amount it increases capabilities relevant for takeover. It’s also unclear how much acceleration in beneficial capabilities is worth a given amount of acceleration in harmful capabilities. I think it would be useful to extend the type of analysis in this post to form a transparent argument for or against the CRC team’s work overall.

Thank you to Jadyn Malone, Alex Kastner, and Isha Harris for useful feedback and discussion.