analiza aiqbench
L$^3$ introduces a static token-based routing architecture that generalizes embedding tables to decoder layers, effectively balancing memory and compute without the training instability or hardware inefficiencies often associated with dynamic MoE routing. By caching context-dependent information directly within embeddings, this approach enables fast CPU-offloaded inference while outperforming both dense models and iso-sparse MoEs in language modeling tasks. The technique offers a systems-friendly alternative for scaling sparsity, particularly relevant for researchers optimizing transformer efficiency and inference costs.
L$^3$: Large Lookup Layers
arXiv:2601.21461v3 Announce Type: replace Abstract: Modern sparse language models typically achieve sparsity through Mixture-of-Experts (MoE) layers, which dynamically route tokens to dense MLP "experts." However, dynamic hard routing has a number of drawbacks, such as potentially poor hardware efficiency and needing auxiliary losses for stable training. In contrast, the tokenizer embedding table, which is natively sparse, largely avoids these issues by selecting a single embedding per token at the cost of not having contextual information. In this work, we introduce the Large Lookup Layer (L$^3$), which generalizes embedding tables to model decoder layers as a means of further scaling sparsity. L$^3$ layers use static token-based routing to aggregate a set of learned embeddings per token in a context-dependent way, allowing the model to efficiently balance memory and compute by caching information in embeddings. L$^3$ has two main components: (1) a systems-friendly architecture that allows for fast training and CPU-offloaded inference with no overhead, and (2) an information-theoretic embedding allocation algorithm that effectively balances speed and quality. We empirically test L$^3$ by training transformers with up to 2.6B active parameters and find that L$^3$ strongly outperforms both dense models and iso-sparse MoEs in both language modeling and downstream tasks.
Abstract:Modern sparse language models typically achieve sparsity through Mixture-of-Experts (MoE) layers, which dynamically route tokens to dense MLP "experts." However, dynamic hard routing has a number of drawbacks, such as potentially poor hardware efficiency and needing auxiliary losses for stable training. In contrast, the tokenizer embedding table, which is natively sparse, largely avoids these issues by selecting a single embedding per token at the cost of not having contextual information. In this work, we introduce the Large Lookup Layer (L$^3$), which generalizes embedding tables to model decoder layers as a means of further scaling sparsity. L$^3$ layers use static token-based routing to aggregate a set of learned embeddings per token in a context-dependent way, allowing the model to efficiently balance memory and compute by caching information in embeddings. L$^3$ has two main components: (1) a systems-friendly architecture that allows for fast training and CPU-offloaded inference with no overhead, and (2) an information-theoretic embedding allocation algorithm that effectively balances speed and quality. We empirically test L$^3$ by training transformers with up to 2.6B active parameters and find that L$^3$ strongly outperforms both dense models and iso-sparse MoEs in both language modeling and downstream tasks.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2601.21461 [cs.LG]
(or arXiv:2601.21461v3 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 207 cuvinte
Subiecte conexe
LookME: Lookup-Based Multimodal Embeddings for Layer Injection in Vision-Language Models
arXiv · 21 iul. 2026
How Generative Recommenders Are Redefining RecSys at Scale
NVIDIA · 20 aug. 2026
Challenge: Hand coding weights for efficient sequence memorisation
AI Alignment Forum · 23 iul. 2026
How do neural networks regress cubic polynomials? Apparently, they use a trick invented in Milan 500 years ago
LessWrong · 22 iul. 2026