The imposters among us: function vectors that ace every check and do the wrong task (in search of circularity)

TL;DR: We extracted shift-by-k-months function vectors on Llama-3.2-3B from few-shot prompts that contained fewer distinct months (lower diversity). The vectors passed three classic checks: the behavioral gate, stability when extracting from disjoint halves of the prompt samples (cosine similarity ≥  0.99 for the broken vectors, 0.98 for the full set), and the causal effect (where injection tripled zero-shot accuracy and the correct answer’s probability, including for the most broken vector). However, they encoded a completely different task: output a month adjacent to the queried one, while completely ignoring “k”. Margins ranged from -0.31 to -0.94 across the broken sets, with -1.000 for the most broken set, on the months included in the few-shot prompts.   The culprit is the number of distinct example inputs. Lower diversity makes the model perform the few-shot task better (from 0.38 to 0.83) even though the function vector changes identity, so the checks actually favor the imposters.  After a sweep across layers and strengths, no injection setting (0 out of 672) rescued the broken vectors. Thresholds calibrated on generated random data were wrong in both directions (145–156× too low or 100× too high), compared to the thresholds from shuffled real vectors, which were correct. To catch imposters, report example diversity, and score not only whether injection helps on average but also which function it performs.  Repo: https://github.com/star2vec/tarcle What's in the box? First, you pick it up, and it has weight to it. So something truly is inside, got it. After gently shaking it, you hear a rolling noise, followed by a soft clink whenever the mystery object collides with the box’s walls. Finally, you blindly reach inside and feel the distinct, cold touch of glass. You figure out that the thing inside is somewhat pear-shaped, and the narrower end is made of a different material, and it clicks. “It’s a light bulb!” you say confidently, and eagerly lift the lid to be proven right… But you couldn’t be more wrong. Inside is actually one of those ship-in-a-bottle decorations, with a message attached. Even after multiple verifications, you can still be confidently wrong. To validate function vectors in LLMs, researchers play this exact game, up to a point. We confirm the properties, check their effects, and declare their function. But in our confidence, we often leave out actually opening the box. Circles and vectors A circle has a nice, uniform shape, historically referred to as “divine” or “perfect”. Models are also drawn to it when encoding circular data. Days of the week, colours on a wheel, emotions on a circumplex, and months all sit on measured circles in the activation space (Engels et al., Modell et al., Sun et al.). Months are also what our tasks are based on. If you give a model a list of examples that apply an operation, you can extract the vector of the task being demonstrated itself. Features, values, and concepts have vectors in the models’ representation spaces, and so do functions.  Extraction happens through in-context learning examples. Let’s shift a month by 3: Q: March  A: June Q: April A: July Q: November  A:  And if we extract the shift-by-3 vector and inject it into this prompt: Q: February A: ? The output will be “May”. The vector truly carried the intended task. Of course, a single vector can’t really be circular, but there are 12 months, so there are 12 shift-by-k-months function vectors.   But if feature vectors sit on rings, could this apply to function vectors as well? Could the shift-by-12 vector sit close to the one where we shift by zero? We extracted these shift vectors from Llama-3.2-3B through two independent methods: averaging the outputs of the attention heads causally relevant to the operation (Todd’s method) and using the hidden-state readout (Hendel’s method). If the 12 vectors do live on a circle, the distance between a pair of vectors should depend only on the distance of their shifts, and they should wrap like the hour numbers on a clock.  The result: the task vectors did not trace any circle, regardless of the extraction method. However, vectors do vary, as their differences were 9-16 times the noise floor (the difference between two extractions of the same vector based on disjoint halves of the prompt samples, to account for the random selection of prompt noise). All the vectors shared a main component in an 80-95% proportion that dominates the set’s geometry, and the specific number of months rides on top as a correction. Testing for traces of an open helix turned out inconclusive, and another task family failed to qualify (full diagnostics in the repo).   This is consistent with a recent theory (Karkada et al.) that proposes that the circularity of concept vectors is mainly attributed to symmetries in training data. Months as concepts are distributed evenly, but shift-by-k-months operations are not. Mirroring this, shifts by one, two, or eleven months (whose