How to Open Them Up – Part I

TL;DR We suggest an approach to systematization of the mechanistic interpretability research field, which is tailored to our own research goals and tasks. We identified four main tasks we must solve in order to properly explore one chosen concept and its representations inside LLMs: finding the concept’s representation; establishing its causal role in an LLM’s behavior; establishing its necessity; steering the concept's representation in order to change an LLM’s behavior. In this post we explore approaches and methods applicable to task 1 – finding the concept. We analyzed the available literature and decided to focus on the following ones: linear probes; difference-in-means; sparse autoencoders (SAEs); PCA & clustering. Each has advantages and limitations, and we shared our own experiences and thoughts. In summary, linear probes and difference-in-means, which are somewhat similar, are fairly cheap in terms of computational resources needed, but they’ve been criticised for the lack of precision as well as for the lack of a mechanism that would highlight causal relationships between concepts / concepts and behaviors. To analyze causality, researchers suggested SAEs. They seem to be on the rise recently, although there are certain critiques showing that we must at the very least treat them carefully. All the methods listed above require a concept, defined in advance. PCA and clustering do not – they allow us to extract meaningful components from activations first, then define what we’ve found. Therefore, we decided to include these methods in the post as well. Feel free to jump to any part you’re most interested in using headings. And in the next posts we’ll discuss other tasks and their available methods and approaches. Brief Intro We’re exploring the concept of refusal and the ways refusal is represented in LLMs in order to understand how to use it and make LLMs safer and more reliable for both everyday use and critical-domain use (like in medicine, cybersecurity and so on). In our latest update we’ve talked about the components of refusal and their distinguishability. In the post before that we’ve shared our findings on the shape of refusal. We’ve tried many different approaches and ran a ton of experiments, so eventually we decided to press pause, look around and map the field we explore, then proceed more systematically. So, in the following sections we will focus on some of the existing mechanistic interpretability methods for finding concepts; strengths, weaknesses, and applicability of such methods. We base our taxonomy upon our research, the challenges we’ve encountered, and the discoveries we’ve made along the way, so it is not meant to be 100% comprehensive. If you want a more exhaustive read, we suggest you read the paper titled “Mechanistic Interpretability for AI Safety -- A Review”. It is well-written and covers a broader scope of mechinterp tools and approaches. Having that said, let’s define the four jobs we want to perform with vector representations and the methods we are discussing here: Job 1 – find a concept representation. It means “find a vector or a bunch of vectors of some complex geometric shape that define the concept we’re interested in”. “Define the concept” usually means at this point that we change the concept in the input text, and activations we’ve collected change too, so we suspect there is a connection. Proving causality is the next step we won’t discuss in depth today. Job 2 – establish its causal role. That is, establish if whatever we’ve discovered at the previous step actually changes our LLM’s behavior in a certain way. And describe what the “certain way” is. Job 3 – establish its necessity. This part is closely related to the previous one, but here we want to know whether the representation we’ve found is one of the many factors, a core factor, or a sole factor affecting the behaviour. Job 4 – steer it to affect the behaviour. The crucial step – we influence the representation to make our LLMs more reliable and safe for humans. Further we will discuss linear probes, difference-in-means, sparse autoencoders and PCA / clustering. The order is somewhat random – this is just the way we’ve discovered and tried them out one by one. We are open to a discussion about including other jobs or excluding some of our four and making this list more comprehensive and, more importantly, useful for AI safety research. Finding a representation Linear probes Linear probes are arguably the most popular tool because of their relatively low computational cost. Earliest works on linear probes seem to come from 2016-2018, and this is the first method we’ve came across in our mechinterp journey. “Understanding intermediate layers using linear classifier probes” is one of the earliest papers about linear probes. The authors consider probes “thermometers used to measure the temperature simultaneously at many different locations”. They froze neural networks, extracted activations f