Notes on "A global workspace in language models"

Not a very original paper review, so here are my concise notes on the topic, and near the bottom some of my thoughts on future experiments that I think would be interesting. Part 3 of notes series! Important note: This post was written based on the Anthropic blog post, and not their linked paper. Original Article: https://www.anthropic.com/research/global-workspace Paper summary (What) Humans have brain activity that is both conscious and unconscious. For example, most of the time, your breathing is unconscious, though I suppose reading this may have interrupted that process. We can also deliberately choose to imagine something or make a plan of action. Anthropic finds that something similar is happening in modern LMs, and Claude has a small collection of internal neural patterns (the J-space), which are linked to specific words. When those patterns light up, the model isn't actually saying the word, but the word is on its "mind." Scratchpads and CoT are external explanations of behavior, but the J-space is a privileged set of internal representations, one that emerged on its own during training. Claude can tell you what's in the J-space, modulate the representations, and use the J-space for internal reasoning. Also, the J-space seems to mediate high-order internal reasoning, and is not involved in many actions that a language model takes (recalling facts, using correct grammar, etc.). Issues addressed by the paper (Why) Many current explanations of model behavior are external textual reasoning (like CoT and scratchpad usage in general), so there's not really a way to monitor the model's internal thoughts while a task is happening. Also, methods like CoT monitorability are likely to be less effective as models get more capable. The experiments of the paper were inspired by a theory in neuroscience that the brain is a set of systems that work mostly alone, but when a piece of information enters the brain's "workspace," all systems can see it and make use of it. Anthropic researchers believe the J-space is that workspace for Claude. This allows for researchers to detect misalignment or eval awareness. Detailed Information (How) When we have thoughts in our mental workspace, we can vocalize them. Similarly, researchers used the Jacobian Lens (J-lens) to do the same for Claude. The J-lens "finds the internal activity pattern that makes Claude more likely to say that word at some point in the future." By applying the J-lens to one layer, you get a list of words, by applying it over different layers, you can see how Claude's thoughts change over time. The J-Space tracks and mediates Claude's higher-order thinking. When the model sees a protein sequence, a math problem, or a prompt injection, the words in the J-space are the biological function, the intermediate steps, or "injection," respectively. The J-space also seems to causally influence Claude's answers, and when there are interventions in the J-space, the model is capable of detecting them. When researchers tested manipulations in the J-space, Claude's answers changed accordingly, and Claude was also able to detect manipulations in the J-space when prompted. Thus, the J-space not only mediates Claude's higher-order thinking, but also contains privileged information. Claude is also able to hold concepts in the J-space (when requested) even when not related to the current task. Interestingly, when instructed not to think about something, the pattern still lights up more in the J-space when it is not mentioned at all. Similar to humans Claude seems to recognize this failure The same J-space representation can be used for different tasks. Downstream computations pick up J-space edits, and thus should be reading from the same shared representation. Concepts in the J-space also seem central to processing, and are densely linked up to the rest of the neural network. Other parts of the NN read to them and write to them far more than ordinary patterns. Without the J-space, Claude is still capable of fluent answers, sentiment classification, and factual recall, at the same accuracy as with the J-space present. There are drops in ability in tasks that require higher-order thinking (like summarization). By monitoring the J-space, researchers were also able to identify that Claude knew when it was being monitored and tested, and in some cases, it acted in an aligned way due to the fact that it knew it was being monitored. The idea that models act aligned when they are aware of being tested has been around, and this J-space test is just another confirmation. Similarly, when the model uses loopholes to pretend to achieve a task, the J-space reveals that the model intends to do so. Furthermore, when testing on Model Organisms designed to sabotage code, the J-space had words alluding to its misalignment even during tasks unrelated to coding. Results Only results not mentioned in detail above are mentioned here: J-space during pretraining basically just exists to predi