Fork Around and Find Out Part 3: Interpreting the knight auditor

Summary This is the third and final post in a series detailing our attempts to mechanistically interpret one head of the chess transformer Maia-3. We find that previous causal evidence gave us a spurious picture of the head, and that rather than just detecting knight forks, head 5 of layer 5 (l5h5) predominantly serves to suppress pointless knight moves. Maia-3 architecture from (arxiv.org/pdf/2605.19091) In this final post, we show that intentionally crafted probes shatter this interpretation of l5h5, and what we once called a “fork detector” is more akin to a “knight move auditor,” one that grades knight moves by their threat level, and is predominantly defined by its strong suppression of poor, quiet knight moves. This has some interesting mirrors to themes in cognitive science research which will be discussed at the end. Interactive demos for the post can be found here: chessformer_lens demo.ipynb The analysis and visualization library: https://github.com/chessformer-lens/chessformer_lens Recap The terminology we have been using: layer 0–7 — the eight transformer blocks; each contains an attention(sub)layer and an feed forward MLP (sub)layer. residual stream — the shared communication channel that attention and MLP additively read from and write into; the additivity is what makes per-component decomposition meaningful. readout point — the residual stream's content at a given depth. We read at 18 points: the input embedding, the post-attention and post-MLP points of each of the 8 blocks, and the final encoder output. These point are after layer norm. Depth labels in figures read [emb, a0, m0, a1, m1, …, a7, m7, enc] (aN = block N's attention, mN = its MLP). logit — raw output of a machine learning model forward pass before being transformed by an activation function to fit match with the desired predictions logit lens— decode a readout point through the final unembedding as if it were the finished product, and read the resulting move distribution. decodable move— a move scoring highly under the logit lens at some depth. policy — the network's output distribution over all 4352 moves, also used for the output of a single move. Maia plays directly from this; there is no search. ontology — the model's internal world and how it carves what it knows into categories Our model: Maia-3 is a chessformer model built to mimic human play across skill levels, and thus, we hypothesize, internally represents human recognizable features like knight forks. Each square maps to one token. Its attention heads combine two schemes: standard "semantic" QK self-attention, and a "geometric" GAB scheme carrying positional structure. GAB heads are generated per position by a a small generator network. The running example we have been using is the following position and knight fork Ne6+: Position and policy at 2400 ELO and Win Draw Loss evaluation for White Last post we discovered that knight forks become decodable at layer 5 of Maia-3, and by ablation we found that head 5 is unquestionably the causal link: Δ per layer-5 head over N=100 forks: h5 dominates at -2.78 change in logit, whereas every other head ablation changes no more than 0.14 logit. We performed an ablation sweep on each head for 300 random knight forks and again find l5h5 is by far the most causally active among all layers' components. We have found this method to be especially useful for this and similar projects. ------------------------------------------------------------------------------------------------------ Part 3 The first domino falls Although the results so far strongly indicated that we localized a fork computing head and that the model at least partly organizes its ontology by tactics, we opt to run a few more probes, which end up shattering this hypothesis. Since every forward pass above was performed on a knight move, we now consider the identical tactical motif with a different piece, a pawn fork. One mined example below: Upon ablating l5h5, and running N=250 ablations on different such pawn forks, we find that the K+Q fork by a pawn rather than knight has an average logit change of +0.10—the same tactic is essentially unaffected by our head when a different piece is used. The head is not about general forks. In fact, (N=300) pawn forks yield the full ablation sweep heatmap below Sign convention everywhere is Δ = ablated − clean so positive implies a suppressing head. This looks nothing like the sparse knight heatmap with its single dominant carrier head A gradient of threats Next we mine the lichess.org database for non forcing knight moves that attack a defended or worthless target, such as a movable rook, and perform the same ablation on N=400 of each class of attacked piece. This is to begin to untangle any structure in the value of the threat the knight creates. Hollow attacks on pawns are slightly supported, hollow attacks on the more valuable bishops are neural, hollow attacks on the even more valuable rook are suppressed a