analiza aiqbench
This research investigates whether decoder-only language models can effectively learn without a trainable input embedding table, replacing it instead with fixed minimal binary token codes. The study demonstrates that removing approximately 12.5% of the model's parameters via this approach yields perplexity scores broadly similar to standard baselines, suggesting that continuous representations can be learned directly from fixed identifiers in specific regimes. While these checkpoints do not yet outperform larger reference models like SmolLM2, the findings offer a viable pathway for reducing parameter counts and memory footprint without sacrificing core language modeling capabilities.
Language Models Without a Trainable Input Embedding Table: Learning from Fixed Minimal Binary Token Codes
arXiv:2605.09751v2 Announce Type: replace Abstract: We study whether a decoder-only language model requires an independently trainable input vector for every token. For a vocabulary of size $V$, an injective fixed-length binary identifier requires $K=\lceil\log_2 V\rceil$ bits. We replace the usual trainable $V\times d_{\mathrm{model}}$ input table with fixed minimal binary token codes and a parameter-free tiled lift to model width. With $V=65{,}536$ and $d_{\mathrm{model}}=1024$, this supplies each token as a fixed 16-bit code and removes 67.1M trainable parameters, approximately 12.5% of the untied learned-input baseline. We also study a table-free implementation with one fixed invertible affine recoding over $\mathbb{F}_2^{16}$. Across three training seeds, 32-layer models trained on approximately 16-17B tokens obtain mean held-out perplexities of 2.44 for the learned-input baseline, 2.36 for canonical binary codes, and 2.39 for affine-recoded codes. These descriptive results do not establish statistical superiority or equivalence. Standardized evaluation of released base checkpoints with the LM Evaluation Harness adds commonsense, knowledge, and language-modeling benchmarks. The three paper checkpoints show broadly similar, task-dependent performance, while external SmolLM2 reference models are substantially stronger on many tasks. Our conclusion is therefore limited to the studied regime: a free trainable token-indexed input table is not required to learn nontrivial language modeling. The Transformer still learns continuous representations, and the output vocabulary projection remains standard and trainable.
Abstract:We study whether a decoder-only language model requires an independently trainable input vector for every token. For a vocabulary of size $V$, an injective fixed-length binary identifier requires $K=\lceil\log_2 V\rceil$ bits. We replace the usual trainable $V\times d_{\mathrm{model}}$ input table with fixed minimal binary token codes and a parameter-free tiled lift to model width. With $V=65{,}536$ and $d_{\mathrm{model}}=1024$, this supplies each token as a fixed 16-bit code and removes 67.1M trainable parameters, approximately 12.5% of the untied learned-input baseline. We also study a table-free implementation with one fixed invertible affine recoding over $\mathbb{F}_2^{16}$. Across three training seeds, 32-layer models trained on approximately 16-17B tokens obtain mean held-out perplexities of 2.44 for the learned-input baseline, 2.36 for canonical binary codes, and 2.39 for affine-recoded codes. These descriptive results do not establish statistical superiority or equivalence. Standardized evaluation of released base checkpoints with the LM Evaluation Harness adds commonsense, knowledge, and language-modeling benchmarks. The three paper checkpoints show broadly similar, task-dependent performance, while external SmolLM2 reference models are substantially stronger on many tasks. Our conclusion is therefore limited to the studied regime: a free trainable token-indexed input table is not required to learn nontrivial language modeling. The Transformer still learns continuous representations, and the output vocabulary projection remains standard and trainable.
Computation and Language (cs.CL)
arXiv:2605.09751 [cs.CL]
(or arXiv:2605.09751v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2605.09751 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 302 cuvinte
Subiecte conexe
Do Language Models Need a Trainable Input Embedding Table? Fixed Minimal Token Codes at 1.7B-Class Scale
arXiv · 6 oct. 2026
Training Azerbaijani language models on Amazon SageMaker AI
AWS · 28 mai 2026
Challenge: Hand coding weights for efficient sequence memorisation
AI Alignment Forum · 23 iul. 2026
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · 8 oct. 2026