Confirming Claims of Superposition and Adversarial Examples in Toy Models

This is a replication of Adversarial Attacks Leverage Interference Between Features in Superposition, completed as part of the Second Look Summer Fellowship.

We reproduce all three core claims of Stevinson et al. from their toy classifier setting: PGD attacks against toy models generally agree with theoretically optimal solutions. Toy models without superposition are less vulnerable to attacks. Robustness falls monotonically as superposition increases. Attacks transfer between independently-trained models because feature geometries can converge across training runs.

PGD attacks against toy models generally agree with theoretically optimal solutions.

Toy models without superposition are less vulnerable to attacks. Robustness falls monotonically as superposition increases.

Attacks transfer between independently-trained models because feature geometries can converge across training runs.

Similar to what's reported by Elhage et al., 2022, we notice that co-activating features can collapse onto a shared direction instead of orthogonalizing.

Our code can be found here and we encourage readers to fork/run followup experiments!

Elhage et al., 2022 defines superposition as the phenomenon where "models represent more features than they have dimensions." To study this in a toy setting, they define a classifier where features are squeezed into an m-dimensional bottleneck with , so feature directions interfere in the latent space.

Stevinson et al., the origin of the experiments in this replication, argues this interference is a mechanism behind adversarial examples. The paper makes three claims: 1. adversarial attacks exploit interference, 2. models without superposition are more robust to adversarial examples, 3. correlations in the input data allow models to converge to similar geometry allowing adversarial attacks to transfer between models.

The experiments in the paper deal with a toy model with encoder and decoder . For input the output is . Here is the number of classes and is bottleneck width forcing the features to be condensed into a lower dimensional space before reconstruction. This gives us a convenient architecture to study superposition: a large ratio means the model has to do more compression, while if , the model can represent each feature with its own direction.

The model uses cross entropy loss. Features are sampled independently with probability , where s can be thought of as how sparsely features are activated within the data.

Adversarial attacks are computed by projected gradient descent (PGD) (Madry et al., 2019). For a given sample , the adversarial example is found by PGD, where indicates how the attack modified the original sample. measures distance between the original example and the adversarial example and ε is the attack budget which enforces that . We use PGD learning rate , at 50 steps with 5 restarts, and . The values for k and m vary but unless otherwise stated and .

We follow the hyperparameters described in the paper as of July, 2026.[1]

Stevinson et al. hypothesize that adversarial attacks in toy models work by exploiting the interference geometry. Proposition 1 from the paper shows the optimal perturbation has a closed form solution:

where ws, wt are the source/target class vectors (rows of the decoder Wd) and We the encoder, scaled to the ε-ball. The authors test if the PGD perturbation matches the optimal attack derived from interference geometry, empirically by:

Performing untargeted attacks on the toy model.

Taking the set of successful attacks (an attack is successful if the prediction flips and the true label stays unchanged).

Finding the optimal targeted attack for target class using the equation above and calculating the cosine similarity with .

At a high level, this can be thought of as letting the attack maximize loss in an arbitrary direction and testing if it finds an optimal direction for a given target class. Figure 1 shows the untargeted attack live where and . Each arrow is one PGD step in the encoder's latent plane, the dashed outline is the reachable set (the encoder image of the -ball), and the shaded regions show which class the decoder assigns (the attack succeeds when the activation is pushed across a decision boundary).

We use untargeted attacks for these experiments to follow the methodology of the paper, but we also repeat this experiment for targeted attacks with results in Appendix A.

The claim that the PGD attack aligns with the optimal attack, exploiting the interference geometry, replicates cleanly on the toy model. With sparsity 0.98, we demonstrate that PGD attacks align with optimal perturbation. We believe our cosine similarity results are higher since we let PGD iterate to find the best perturbation while the paper stops at the first prediction flip.

Like the original paper, we ablate on randomness by calculating the cosine similarity between our optimally successful PGD attack and a random direction within -ball, and confirm on average that the cosine similarity between random directions and our PGD attack lands around 0.

As an additional stress test, we repeat our ablation while only selecting random directions that result in a successful attack, isolating the comparison between the PGD attack and random, successful attacks. These can yield non-zero cosine similarity scores, but the PGD attacks still align better with the optimal solutions.

Stevinson et al. claim that toy model adversarial vulnerabilities disappear along with superposition. We observe the same trend with robust accuracy falling monotonically as superposition pressure rises. When (all features can be orthogonal), robust accuracy grows to 97.3%.

Some of the magnitudes we report diverge. At m = 2 our model stays more robust than the paper's because we train for 20k steps while the paper only trains for 5k steps. At our robust accuracy is lower than the paper since we give PGD random starting positions, bigger steps, and 5 restarts to find a better attack, which is a setting stated in the paper but missing in the authors' code.

Stevinson et al. claim that input correlations shape the learned feature geometry, and attacks transfer between independently-trained models to the degree their geometries converge.

Global: adjacent classes frequently co-activate, specifically: Each sample draws a phase ; class c activates with probability . Active classes get Uniform(0,1) values; the prediction label is given by argmax. This process makes index-neighbors more likely to activate together. This incentivizes geometry to converge across seeds.

Each sample draws a phase ; class c activates with probability .

Active classes get Uniform(0,1) values; the prediction label is given by argmax.

This process makes index-neighbors more likely to activate together. This incentivizes geometry to converge across seeds.

Figure 2: Illustration of the global correlation regime.

Figure 2: Illustration of the global correlation regime.

We see the same trend where both geometric similarity and attack transfer rate grow as the correlation of input increases.

Figure 3: Geometric similarity and targeted attack transfer rate per regime, ours vs paper.

Figure 3: Geometric similarity and targeted attack transfer rate per regime, ours vs paper.

Figure 4 shows the geometry between the two seeds. As expected, uncorrelated arrangements are random to each other and unrelated between models.

For the other two settings, we observe an interesting phenomenon: correlated features tilt toward each other rather than orthogonalizing. We dive into it in the next section.

Figure 4: Class-vector geometry for two seeds across the three regimes. In (b) co-activating partners share a color and a collapsed pair is haloed; in (c) the dashed loop links the cyclic index-neighbors.

Figure 4: Class-vector geometry for two seeds across the three regimes. In (b) co-activating partners share a color and a collapsed pair is haloed; in (c) the dashed loop