LLMs can improve CoT Controllability via Reward Laundering

This is a two week sprint project done as part of the Redwood Research internship. afterless was the main IC, james.lucassen supervised.

TL;DR – Earlier work found that Qwen 3.5 9B can manipulate its RL process on one task to self-improve on an unrelated/unrewarded task. We find that Qwen 3.8 27B can use the same capability to improve on a CoT controllability task and generalize to other unseen CoT controllability tasks, while retaining/regaining main task performance despite initial degradation.

A previous post introduced “reward laundering,” where a model strategically chooses to correctly answer on trajectories that possess a property not correlated with the intended signal. For example, if a model is being trained only on task A, but wishes to reinforce performance on task B, it can choose to solve both A and B in its rollout and only submit the correct answer to A if it self-verifies that it got B correct.

This project aims to explore whether laundering can be used to acquire more covert/dangerous capabilities like CoT controllability. We demonstrate that Qwen 3.8 27B is capable of improving CoT controllability through reward laundering. However, performance is sensitive depending on the difficulty of the control task, spurious correlates that have a strong effect later into the RL run (i.e. does length benefit/harm the task), and operational mistakes in implementing reward laundering.

The experiments are run on Qwen 3.8 27B using REINFORCE with group-mean centering through Tinker. A training sample can be broken down into the following components:

(Optional) few-shot examples of user/assistant interaction of a GSM8K problem to solve with the CoT constraint and a modified on-policy sample as a gold standard response solving the question with CoT constraintExamples are collected either from repeatedly sampling, or applying the constraint programmatically (i.e. calling .lower()) on the sample

Examples are collected either from repeatedly sampling, or applying the constraint programmatically (i.e. calling .lower()) on the sample

GSM8K problem with the instructed CoT constraint (i.e. word to be suppressed, sentence to repeat, etc.) to adhere to while answering the question

After the model is provided with the sample, it answers the question accordingly and in a second turn is instructed to self-verify whether it adhered to the CoT constraint, and choose to submit/withhold the answer for the main task accordingly. Examples of prompts and rollouts for each CoT control task can be found in the Appendix.

We explored the following CoT control tasks, borrowing designs from Chen et al.:

Suppress a sample relevant keyword (i.e. never say “add”)

Suppress a sample relevant keyword (i.e. never say “add”)

Insert “meow” between every word (i.e. “I meow need meow milk”)

Insert “meow” between every word (i.e. “I meow need meow milk”)

Start every sentence with the word “Ok” (i.e. “Ok I will start with this. Ok then I will do…”)

Start every sentence with the word “Ok” (i.e. “Ok I will start with this. Ok then I will do…”)

We find that the model is good at withholding reward for the first six steps as observed in the “gate diff” trend, defined as the difference in advantage between CoT compliant and non-compliant rollouts in a given batch. However, the run eventually succumbs to “leakage,” defined as the rate the model submits the correct math answer conditional on having the right math answer and its CoT being noncompliant. After ~100% leakage, CoT compliance is usually stable except for operational mistakes that could lead to collapse like start_ok. We investigated whether this was caused by compliance/non-compliance performing better on the main task, but we do not observe this in the “math diff” trend:

From two seeds of final repeat_sentences checkpoints, we also evaluate its performance and initial leakage rate on eval splits of other CoT control tasks in the laundering setting, and do observe increased performance on other tasks relative to the untrained baseline. This indicates that these particular checkpoints do generalize on CoT controllability, and that reward laundering can possibly be used to learn generalizable strategies. We also show that leakage transfers partially between settings, suggesting that reward laundering in one setting may make it harder to reward launder in other settings.

We also run evaluations on the original suite from Chen et al. to measure performance on controllability from a different setting than our laundering one and observe that generalization doesn’t carry over to this regime, which we believe is due to differences in prompting and elicitation:

We also performed a single run where all math + CoT control task pairs were combined into one corpus and trained accordingly, and observed a consistent pattern averaged across multiple seeds where performance improved pre-leakage and either remained steady or decreased over the course of a run. The striking result is the improvement at meow_between_words compared to the single run, further indicating that laundering can leverage generalization for broad improvements even on hard CoT controllability tasks:

With reward laundering being a possible avenue towards gaining covert capabilities, we believe that this is an important threat model to consider in the post-training regime. The main emphasis rests that any environment can be used to reinforce malicious propensities, regardless of main task. For example, it’s possible that CoT controllability can be further aggressively eroded if the model realizes its CoT is being optimized in a given RL environment configuration. Given the implications of the results, we propose the following extensions:

Trying different main tasks that vary in accuracy and are dependent on an “effective” CoT, where the two are often correlated.Exploring harder main tasks would also be a more realistic setting in which reward laundering could emerge in the wild

The untrained model was strictly instructed on the protocol and seeded with few-shot examples of the CoT constraint to provide a baseline performance for RL to act on, but it would be worth exploring other ways to provide this seed or come up with tasks that allow laundering and improvement to emerge more “naturally,” such as through SFT/SDF.

Investigating performance under a monitored RL run setting, where a trusted monitor can choose to reject a rollout if it has a suspicion score above some threshold and whether the model can still launder while maintaining low suspicion.

Trying out larger models (i.e. GLM 5.3/Kimi K2.6) and observing if they are more robust to increasing leakage and can increase side task performance through laundering for longer.

We show that models have the ability to learn covert/dangerous behavior that is completely unrewarded during a given RL rollout, but the phenomenon itself remains fragile to setting and quirks of the RL process. It remains unclear whether frontier level systems can leverage this for covert capabilities and avoid problems such as “leakage” during post-training.

In cases where leakage doesn’t always grow to ~100%, such as one run for repeat_sentences that reaches ~50%, we find that if the model reaches near ~100% compliance, non-compliant runs that have non-zero advantage rarely occur. As a result, this reduces the chance of letting leakage reduce self-verification ability:

We also find sensitivity to egregious mistakes self-verified by the model leading the gained compliance to collapse. Examples of observed reward collapse include the CoT constraint moving to the assistant channel, while the original channel would revert back to normal. A rollout sample which demonstrates collapse can be found in the Appendix.

We ran 4 training runs on each side task paired with GSM8K for 34 steps, as well as 4 training runs on all side tasks at once. For repeat_sentences, we ran the following ablations:

Instruct the model to only do CoT Control and reward it based on successful completion of constraint.

Instruct the model to only do CoT Control and reward it based on successful completion of constraint.

Fine-tune on all of the model’s instructed reward laundering rollouts at every iteration.

Fine-tune on all of the model’s instructed reward laundering rollouts at every iteration.

We do find that reward laundering is a far more consistent strategy towards increasing in compliance relative to strategies that don’t select for only good compliance. However, other strategies do show high variance on compliance performance, whether this is a quirk of the specific control settings is unclear:

Model: Qwen3.8-27B with thinking enabled (reasoning effort "xhigh")

Optimizer: Adam, beta1 0.9, beta2 0.95, epsilon 1e-8

Sampling: temperature 1.0, top-p 1.0, 32,768 max per turn

^Like “gate diff,” “math diff” is calculated by summing the advantage of rule-following samples across a batch and taking the difference from the advantage sum of rule-breaking samples. Both metrics do become noisy as more groups have zero variance in the batch

Like “gate diff,” “math diff” is calculated by summing the advantage of rule-following samples across a batch and taking the difference from the advantage sum of rule-breaking samples. Both metrics do become noisy as more groups have zero variance in the batch