aiqbench analysis
This research investigates whether large language models utilize functional subspaces and vector algebra to solve in-context learning tasks, moving beyond simple linear directions to structured geometric operations. By analyzing residual streams, the study provides empirical evidence that LLMs can accumulate evidence within specific subspaces to perform complex reasoning without explicit training on those skills. This work deepens our understanding of the internal mechanisms driving emergent abilities in foundation models.
Functional Subspace, where language models can use vector algebra to solve problems
arXiv:2602.01687v3 Announce Type: replace-cross Abstract: Large language models (LLMs) were invented for natural language tasks such as translation, but they have proved that they can perform highly complex functions across domains. Additionally, they have been thought to develop new skills without being trained on them. These learning capabilities lead to LLMs adoption in a wide range of domains. Thus, it is imperative that we understand their operating mechanisms and limitations for proper diagnostics and repair. The earlier studies proposed that high level concepts are encoded as linear directions in LLMs activation space and that the geometry of embeddings have semantic meanings. Inspired by these studies, we hypothesize that LLMs may use subspaces and vector algebra in subspaces to perform tasks. To address this hypothesis, we analyze LLMs' functional modules and residual streams collected from LLMs engaging in in-context learning (ICL), one of the emergent abilities. Our analyses suggest that 1) LLMs can create subspaces, where evidence can be accumulated and 2) ICL tasks can be solved via simple algebraic operations in subspaces.
Abstract:Large language models (LLMs) were invented for natural language tasks such as translation, but they have proved that they can perform highly complex functions across domains. Additionally, they have been thought to develop new skills without being trained on them. These learning capabilities lead to LLMs adoption in a wide range of domains. Thus, it is imperative that we understand their operating mechanisms and limitations for proper diagnostics and repair. The earlier studies proposed that high level concepts are encoded as linear directions in LLMs activation space and that the geometry of embeddings have semantic meanings. Inspired by these studies, we hypothesize that LLMs may use subspaces and vector algebra in subspaces to perform tasks. To address this hypothesis, we analyze LLMs' functional modules and residual streams collected from LLMs engaging in in-context learning (ICL), one of the emergent abilities. Our analyses suggest that 1) LLMs can create subspaces, where evidence can be accumulated and 2) ICL tasks can be solved via simple algebraic operations in subspaces.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2602.01687 [cs.CL]
(or arXiv:2602.01687v3 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2602.01687 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 263 words
Related coverage
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · Oct 8, 2026
Lattice-Based Cryptography Explained
Quantum Zeitgeist · Aug 15, 2026
AlgoTrace: Algorithmic Primitives and Compositional Geometry of Reasoning in Language Models
arXiv · Aug 27, 2026
From Vector Search to Agentic RAG: Building an Enterprise Research Analyst with hipVS
AMD · Jul 15, 2026