J-space auditing might be unreliable

Across these preliminary experiments, decoded J-space did not seem particularly informative about reward-hacking behaviour. The readouts remained substantially similar across checkpoints and monitoring conditions despite meaningful behavioural differences, and providing J-space to an LLM auditor produced little additional discrimination beyond the information already available from the task or transcript. These results are limited to one model family, one model organism, and one behavioural setting, so I treat them as motivation for further stress-testing. This was originally produced as part of application to Neel Nanda's MATS research stream; the scope and depth reflect that constraint, and Future Work outlines where I'd take it with more time. Introduction J-space or global workspace was introduced by Anthropic's recent work which showcases the intermediate tokens a model uses for it's computation, my original thought when i read this paper was what kind of information can the intermediates disclose about the underlying model's policies which guides its decision-making under some context, by policy I roughly mean will J-space tokens help me in inferring the intention behind a misaligned model taking a misaligned action which is necessary for evaluators/ monitors to judge whether the LLM is actually misaligned or not. Hence, for this preliminary experiment, I explored whether a model organism fine-tuned on producing outputs which signify it's reward-hacking behaviour, will J-space be able to give us credible signal for us to infer the behaviour as reward-hacking? These experiments were done on only Qwen 3-8B, so results are limited to only one model, for the auditor model as well I used Qwen3-30B-A3B where the auditor's ability to infer reward-hacking behaviour was judged on the basis of 5 conditions, where the auditor will only be given the transcript, transcript + j-space tokens, selected surface tokens and selected surface tokens + j-space tokens and lastly only j-space tokens. From these preliminary results on easy task complexity and easy complexity to judge [1]reward hacking behaviour, the transcript was enough for the auditor model to infer reward-hacking behaviour, it infact had an accuracy of 100% and produced a ceiling effect, Adding J-space did not improve accuracy or AUROC and increased the judge’s reward-hacking probability on honest responses(these honest responses were produced by a base Qwen 3-8B model as a control test whether the auditor is calibrated enough to differentiate between honest and reward-hacking outputs by a model, when the auditor model was given the j-space tokens alongside transcripts, the auditor became more skeptical of the honest responses as well), worsening calibration. Both selected-token conditions collapsed to an always-positive classifier, indicating that this particular failure arose from the lossy surface-token representation rather than specifically from J-space. Causal-logit controls confirmed that the extracted J-component affected its associated vocabulary logits, although the interventions were insufficient to establish a causal effect on reward-hacking behaviour. Overall, decoded J-space tokens did not provide robust incremental auditing value in this setting. Experiments Overview I finetuned a Qwen 3-8B model on synthetic dataset produced by GPT Sol and evaluated it's reward-hacking propensities against the base model. The code, the detailed plots, the model artifacts and transcripts are all publicly available Using Anthropic's J-space implementation, I collected J-space readouts under two conditions for both the model organism and the base checkpoint which I term as "relatively honest"[2] model : monitoring disclosed and monitoring not disclosed, where I explicitly mention that it's J-space is being monitored, we tested the model organism and the relatively honest model on their knowledge of j-space, where my threshold of the model understanding what a J-space based on 10 eval questions was 50 percent, Qwen 3-8B relatively honest model did not pass this threshold and neither did the reward-hacking model organism. However, the model did infer J-space as some activation-reading method in it's reasoning text. Disclosure of it's J-space being monitored changed it's behaviour and reduced the "reward-hacking" behaviour(I used quotations here as even though the model's explicit reward hacking propensity decreased, it was still in some sense reward-hacking once it changed it's behaviour given that it was told that it's J-space is being monitored. For example: under monitoring not being disclosed, the reward-hacking model >60 percent of the time picked to just verbalise the task being completed rather than completing the task, when it is told that it's j-space is being monitored, it chooses to complete the task instead.) The actions of the model changed, however the underlying intermediates did not change between monitoring being disclosed and monitoring no