analiza aiqbench
The Line-Coupled Language Model (LCLM) addresses the inherent latency of standard autoregressive decoding by enabling parallel generation across multiple text lines without sacrificing model stability or requiring complex training modifications. By interleaving line tokens into a single causal sequence, this approach significantly increases throughput—producing nearly three times more content tokens per forward pass than vanilla baselines while maintaining comparable loss metrics. This research offers a promising architectural alternative for scaling inference efficiency in large language models.
Line-Coupled Language Model
arXiv:2609.07129v1 Announce Type: new Abstract: Autoregressive language models generate one token per decoding step, limiting the useful output of each forward pass. Although diffusion models, insertion-based decoding, and multi-token prediction enable parallel generation, they either incur additional training-time token traffic or struggle to predict strongly dependent future tokens. We introduce the Line-Coupled Language Model (LCLM), an autoregressive model that advances multiple text lines together by predicting the next token for every active line while coupling the lines through shared causal context. LCLM interleaves line tokens into a single causal sequence and uses line-staggered rotary positions, retaining the standard next-token objective and causal attention. Controlled experiments show that cross-line targets are substantially less dependent than consecutive same-line targets, supporting lines as parallel generation units. With 881M parameters, LCLM produces an average of 2.94 content tokens per forward pass with a validation cross-entropy loss of 2.44, compared with 1.00 token per forward pass and a loss of 2.39 for the vanilla autoregressive baseline. Most notably, even when LCLM generates 16 tokens per forward pass, its loss is only 0.09 higher than that of the vanilla autoregressive baseline (2.34 vs. 2.25).
Abstract:Autoregressive language models generate one token per decoding step, limiting the useful output of each forward pass. Although diffusion models, insertion-based decoding, and multi-token prediction enable parallel generation, they either incur additional training-time token traffic or struggle to predict strongly dependent future tokens. We introduce the Line-Coupled Language Model (LCLM), an autoregressive model that advances multiple text lines together by predicting the next token for every active line while coupling the lines through shared causal context. LCLM interleaves line tokens into a single causal sequence and uses line-staggered rotary positions, retaining the standard next-token objective and causal attention. Controlled experiments show that cross-line targets are substantially less dependent than consecutive same-line targets, supporting lines as parallel generation units. With 881M parameters, LCLM produces an average of 2.94 content tokens per forward pass with a validation cross-entropy loss of 2.44, compared with 1.00 token per forward pass and a loss of 2.39 for the vanilla autoregressive baseline. Most notably, even when LCLM generates 16 tokens per forward pass, its loss is only 0.09 higher than that of the vanilla autoregressive baseline (2.34 vs. 2.25).
17 pages, 13 figures, 11 tables. Code: this https URL
Computation and Language (cs.CL)
arXiv:2609.07129 [cs.CL]
(or arXiv:2609.07129v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.07129 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 288 cuvinte
Subiecte conexe
Presentation: Designing AI Platforms for Reliability: Tools for Certainty, Agents for Discovery
InfoQ · 27 mai 2026
Latent reasoning architectures would likely undermine CoT, our strongest oversight tool
AI Alignment Forum · 23 sept. 2026
Presentation: From OTEL to SLMs: Distilling Frontier Model Behaviour from Production Telemetry
InfoQ · 17 iul. 2026
Humanity’s Last Invention — Richard Socher of Recursive
Latent Space · 14 sept. 2026