aiqbench analysis
This paper introduces a weights-only diagnostic method to distinguish genuine concept subspaces from dominant output readout directions in large language models. By rigorously testing across 26 models and multiple estimators, the authors demonstrate that specific reasoning concepts occupy minimal energy relative to final-layer PCA or next-token controls, challenging assumptions about how representations map to outputs. The study provides a robust framework for validating concept extraction without relying on activation data alone, advancing interpretability research in machine learning.
Concept Subspaces Compute Beyond the Logit Lens: A Weights-Only Test for Locating Representations Upstream of Readout
arXiv:2609.39263v1 Announce Type: new Abstract: A concept subspace's effect on model behavior does not establish how it relates to the output readout. We introduce a two-sided geometric diagnostic that measures an extracted subspace's overlap with the dominant right-singular directions of the unembedding matrix, evaluated against output-oriented positive controls. Given an extracted basis, the raw diagnostic requires only model weights. Our testbed is the Format-Agnostic Reasoning Subspace (FARS), a ten-dimensional basis extracted from eighteen reasoning concepts expressed in six surface forms. Across nine rank-matched estimators and twenty-six models, four activation-derived concept estimators carry only 0.38--0.80% mean energy in the top-ten readout span. Final-layer PCA carries 3.56%, exceeding FARS in 25 of 26 models. A same-layer next-token control, evaluated using a fitted linear translator for depth matching, carries approximately thirteen times more energy than FARS, with separation in all 25 tested models. Re-extracting FARS on ten disjoint concepts yields 62--100% cross-format retrieval across twenty-four generative models, demonstrating transfer of the extraction procedure rather than a fixed basis. A complementary four-model, three-seed intervention study finds model-dependent source-directed effects that remain well below full-vector replacement. Together, the geometry and intervention controls distinguish concept structure from dominant readout directions while limiting claims of causal sufficiency.
Abstract:A concept subspace's effect on model behavior does not establish how it relates to the output readout. We introduce a two-sided geometric diagnostic that measures an extracted subspace's overlap with the dominant right-singular directions of the unembedding matrix, evaluated against output-oriented positive controls. Given an extracted basis, the raw diagnostic requires only model weights. Our testbed is the Format-Agnostic Reasoning Subspace (FARS), a ten-dimensional basis extracted from eighteen reasoning concepts expressed in six surface forms. Across nine rank-matched estimators and twenty-six models, four activation-derived concept estimators carry only 0.38--0.80% mean energy in the top-ten readout span. Final-layer PCA carries 3.56%, exceeding FARS in 25 of 26 models. A same-layer next-token control, evaluated using a fitted linear translator for depth matching, carries approximately thirteen times more energy than FARS, with separation in all 25 tested models. Re-extracting FARS on ten disjoint concepts yields 62--100% cross-format retrieval across twenty-four generative models, demonstrating transfer of the extraction procedure rather than a fixed basis. A complementary four-model, three-seed intervention study finds model-dependent source-directed effects that remain well below full-vector replacement. Together, the geometry and intervention controls distinguish concept structure from dominant readout directions while limiting claims of causal sufficiency.
54 pages. Substantially revised preprint: new title, expanded model coverage, readout-geometry controls, supplementary intervention and transfer experiments, revised interpretation, updated figures and author list
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2609.39263 [cs.CL]
(or arXiv:2609.39263v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.39263 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 241 words
Related coverage
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · Oct 8, 2026
Reflections on unlearning and inoculation
LessWrong · Sep 20, 2026
Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face
AI Alignment Forum · Aug 3, 2026
Does DiffusionGemma do latent reasoning?
AI Alignment Forum · Aug 16, 2026