analiza aiqbench
This research proposes a parameter-efficient architecture for LLM attention layers by decomposing linear transformations into geometric primitives using Clifford algebra. By representing layers as compositions of bivectors and rotors, the method achieves performance comparable to strong baselines like block-Hadamard while drastically reducing parameter count from O(d^2) to O(log^2 d). The work offers a novel algebraic perspective on model compositionality, suggesting that deep learning functions may emerge from fundamental geometric building blocks.
Composing Linear Layers from Irreducibles
arXiv:2507.11688v4 Announce Type: replace Abstract: Contemporary large models often exhibit behaviors suggesting the presence of low-level primitives that compose into modules with richer functionality, but these fundamental building blocks remain poorly understood. We investigate this compositional structure in linear layers by asking: can we identify/synthesize linear transformations from a minimal set of geometric primitives? Using Clifford algebra, we show that linear layers can be expressed as compositions of bivectors -- geometric objects encoding oriented planes -- and introduce a differentiable algorithm that decomposes them into products of rotors. This construction uses only O(log^2 d) parameters, versus O(d^2) required by dense matrices. Applied to the key, query, and value projections in LLM attention layers, our rotor-based layers match the performance of strong baselines such as block-Hadamard and low-rank approximations. Our findings provide an algebraic perspective on how these geometric primitives can compose into higher-level functions within deep models.
Abstract:Contemporary large models often exhibit behaviors suggesting the presence of low-level primitives that compose into modules with richer functionality, but these fundamental building blocks remain poorly understood. We investigate this compositional structure in linear layers by asking: can we identify/synthesize linear transformations from a minimal set of geometric primitives? Using Clifford algebra, we show that linear layers can be expressed as compositions of bivectors -- geometric objects encoding oriented planes -- and introduce a differentiable algorithm that decomposes them into products of rotors. This construction uses only O(log^2 d) parameters, versus O(d^2) required by dense matrices. Applied to the key, query, and value projections in LLM attention layers, our rotor-based layers match the performance of strong baselines such as block-Hadamard and low-rank approximations. Our findings provide an algebraic perspective on how these geometric primitives can compose into higher-level functions within deep models.
Machine Learning (cs.LG)
arXiv:2507.11688 [cs.LG]
(or arXiv:2507.11688v4 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 153 cuvinte
Subiecte conexe
Composable Trust Infrastructure for Manufacturing Knowledge Graphs: Cross-System Provenance, Temporal Reasoning, and Decision Traceability
arXiv · 25 aug. 2026
FP8 is All You Need (Part 1): Debunking Hardware FP64 as the HPC Holy Grail (June 13th version)
arXiv · 16 iun. 2026
Steered Not Stirred: Effective Reward Tilting using Surrogate Reward Mixing
arXiv · 21 oct. 2026
Teach and Grow: An Agent-Centered Architecture for General Robot Learning
arXiv · 20 aug. 2026