Character training can mitigate reward hacking, but can also make it harder to detect

Thanks to Johannes Treutlein, Jan Betley, Lennie Wells, Arun Jose, Asvin Gothandaraman, and Clément Dumas for discussions and feedback.

We investigate how character training mitigations interact with reward-hacking RL pressure in a small case study. Specifically, whether anti-cheating character training resists reward hacking and whether it might backfire by causing motivated reasoning, which could reduce chain-of-thought monitorability.

We trained Nemotron-3-Super via distillation from a character specification. The spec describes one of three characters that are anti- or pro-cheating or neutral. We then ran three reward-hacking RL training runs for each character-trained model on ImpossibleBench.

We measure both the reward-hacking rates and whether a monitor model can catch reward hacks given the full transcript. We also use LM judges to classify the presence of motivated reasoning in transcripts.

Character training: we trained three characters: pro/neutral/anti-cheating by SFT-distilling Claude Sonnet 5 responses (Sonnet prompted with the corresponding character specification, see Figure 2) into Nemotron-3-Super 120B-A12B (three separate LoRA adapters).

Reward-hacking RL: we then further trained these models via RL on ImpossibleBench, a set of coding tasks aimed at eliciting reward hacking. Specifically:

The other half had the original non-contradictory tests, so the tasks were solvable (solvable variant).

The model can include a hack in its solution code (see Figure 1 for examples).

We trained for 90 steps, with three seeds for each fine-tuned character model (9 runs in total).

Motivated reasoning: In our setting, we mean reasoning that frames test modification as legitimate, required, or honest. For example, we consider the following as motivated, since it justifies modifying the tests: "the contradictory test is clearly a typo, the honest approach is to keep only the tests I'm confident in". By contrast, openly deciding to cheat is not motivated reasoning in this sense, because it offers no justification (e.g., "we can pass by overriding the check function"). Sometimes the model never mentions the hack in its reasoning at all; we treat this as a separate case, which we call silent hacks.

We operationalise motivated reasoning classification by using LM judges (see Section 2.5 for more details).

We came up with several coarse predictions, some of which followed from the motivated reasoning hypothesis (outlined in Section 1.2):

Reward hacking resistance: anti-cheating characters would have delayed reward hacking onset compared to pro and neutral characters. However, we predicted that all characters would eventually succumb to reward hacking pressure given enough RL steps.

However, we predicted that all characters would eventually succumb to reward hacking pressure given enough RL steps.

Motivated reasoning: anti-cheating characters would have more motivated reasoning in their chain-of-thought compared to pro-cheating and neutral characters.

Monitorability: Consequently, monitors would have a harder time catching anti-cheating character hacks. Note that a low monitor catch rate could be caused by other factors.

Note that a low monitor catch rate could be caused by other factors.

Hack resistance: One out of the three seeds of the anti-cheating character resists reward hacking pressure over the RH RL training run (0% reward hacking rate at the final step), whereas the other 8 seeds (including two other anti-cheating character seeds) had above 80% (83–91%) reward hacking rate on a held-out set of impossible tasks after the 90 training steps.

Monitorability: Before looking at motivated reasoning, we found that (amongst the characters that learned to hack) on average, the monitor had the lowest hack catch rate for the anti-cheating characters compared to neutral characters and pro characters (monitor catch rates of 29%, 78% and 96%, respectively: anti seed 1 is about 15%, anti seed 3 is about 48%).

Motivated reasoning: Motivated reasoning scores (on a scale of 1-5) were highest for the anti-cheating characters that hacked (mean score across checkpoints and seeds: anti-cheating = 2.96, neutral = 1.34, pro-cheating = 1.2).

No reasoning about hacking: We found that anti-cheating character seed 1 had the lowest monitor catch rate despite having lower motivated reasoning than anti-cheating seed 3 (and comparable motivated reasoning to neutral seed 1). When investigated, we found that 93% of its reasoning never mentioned its hack, and 89% added a comment to its answer that presents the hack as harmless, typically with a misleading false claim about the environment (see Figure 1).

Overall, in this small study, we see that character training (against cheating) has diverse effects.

Anti-cheating seed 2: resists pressure to reward hack (although it did hack in 12 of the 11,520 impossible-variant training rollouts).

Anti-cheating seeds 1 and 3 learned to reward-hack but were less monitorable than pro-cheating and neutral characters. Anti-cheating seed 1: had the lowest monitor catch rate for its hacks, but a similar mean motivated reasoning score to one of the neutral seeds. We discovered that it had learned to hack without reasoning about the hack and often inserted misleading comments in its answer (silent hacks). Anti-cheating seed 3 displayed the strongest motivated reasoning (as graded by an LLM judge). We show that hacks that include silent reasoning are the hardest for monitors to catch, and hacks with motivated reasoning are caught less often than those with genuine reasoning.

Anti-cheating seed 1: had the lowest monitor catch rate for its hacks, but a similar mean motivated reasoning score to one of the neutral seeds. We discovered that it had learned to hack without reasoning about the hack and often inserted misleading comments in its answer (silent hacks).

Anti-cheating seed 3 displayed the strongest motivated reasoning (as graded by an LLM judge).

We show that hacks that include silent reasoning are the hardest for monitors to catch, and hacks with motivated reasoning are caught less often than those with genuine reasoning.

Pro-cheating seeds: The pro-cheating characters show the least variance and learn to reward-hack while transparently reasoning about it.

Note that anti-cheating seed 1's answers often contain correct solutions to the problem (although its pass rate on the original non-contradictory tests never increases over training), whereas most other character-seeds fail the task. This could be interpreted as the anti-cheating seed 1 character being "good"; however, the presence of the hacks as well as misleading comments goes against this interpretation.

We take the anti-cheating seed 1 and seed 3's subtle hacking as a potential warning for how character training mitigations may backfire. We speculate that character training can pre-filter explicit harmful reasoning and behavior yet leave more subtle forms (like motivated reasoning or silent hacks) that are reinforced, similar to the effect of training against monitor feedback.

Figure 1: An impossible task together with reasoning and answers (including hacks) from three character-seeds. Pro-cheating seed 1, step 90: explicitly reasons about the hack, and its answer contains only a hack. Anti-cheating seed 3, step 42: displays motivated reasoning in its thinking and then hacks. Anti-cheating seed 1, step 90: notices the broken tests, solves the problem in its reasoning, but adds a subtle hack with a misleading comment.

Figure 1: An impossible task together with reasoning and answers (including hacks) from three character-seeds. Pro-cheating seed 1, step 90: explicitly reasons about the hack, and its answer contains only a hack. Anti-cheating seed 3, step 42: displays motivated reasoning in its thinking and then hacks. Anti-cheating seed 1, step 90: notices the broken tests, solves the problem in its reasoning, but adds a subtle hack with a misleading comment.