analiza aiqbench
SymbolicLight V1 addresses the persistent quality gap in natively trained spiking language models by coupling binary Leaky Integrate-and-Fire dynamics with a continuous residual stream. While achieving over 89% activation sparsity, the model currently exhibits higher energy consumption per token compared to dense Transformers like GPT-2 on standard hardware, highlighting ongoing challenges in efficiency for neuromorphic architectures.
SymbolicLight V1: Spike-Gated Dual-Path Language Modeling at High Activation Sparsity
arXiv:2605.21333v2 Announce Type: replace Abstract: Natively trained spiking language models must preserve information across time while operating through sparse binary activations, a combination that has produced a persistent quality gap relative to dense Transformers. We present SymbolicLight V1, a spike-gated dual-path language model that couples binary Leaky Integrate-and-Fire (LIF) dynamics with a continuous residual stream. Its Dual-Path SparseTCAM mixer combines a first-order exponential-decay state with windowed local attention on the continuous residual stream, followed by a context-conditioned decoding head. We train four 194M-parameter models from scratch on a 3B-token, 10-domain Chinese-English corpus. On a token-weighted held-out set the runs reach PPL 8.88-8.93 (mean 8.904, sample standard deviation 0.019) at more than 89% per-element activation sparsity. Code tokens are 43.7% of that set; the unweighted mean of the ten domain PPLs is 29.38. Under the same corpus, tokenizer, token budget, and hardware, the token-weighted mean is 7.7% above GPT-2 201M (PPL 8.27). Across five zero-shot benchmarks the two 200M-scale models show no clear accuracy separation. Under sampling with temperature 0.7 and top-k 50, SymbolicLight produces lower 4-gram repetition; an entropy-modulated rule reverses that ranking. On a measured RTX 2080 Ti, SymbolicLight uses 2,848 mJ/token versus 905 mJ/token for GPT-2 201M.
Abstract:Natively trained spiking language models must preserve information across time while operating through sparse binary activations, a combination that has produced a persistent quality gap relative to dense Transformers. We present SymbolicLight V1, a spike-gated dual-path language model that couples binary Leaky Integrate-and-Fire (LIF) dynamics with a continuous residual stream. Its Dual-Path SparseTCAM mixer combines a first-order exponential-decay state with windowed local attention on the continuous residual stream, followed by a context-conditioned decoding head. We train four 194M-parameter models from scratch on a 3B-token, 10-domain Chinese-English corpus. On a token-weighted held-out set the runs reach PPL 8.88-8.93 (mean 8.904, sample standard deviation 0.019) at more than 89% per-element activation sparsity. Code tokens are 43.7% of that set; the unweighted mean of the ten domain PPLs is 29.38. Under the same corpus, tokenizer, token budget, and hardware, the token-weighted mean is 7.7% above GPT-2 201M (PPL 8.27). Across five zero-shot benchmarks the two 200M-scale models show no clear accuracy separation. Under sampling with temperature 0.7 and top-k 50, SymbolicLight produces lower 4-gram repetition; an entropy-modulated rule reverses that ranking. On a measured RTX 2080 Ti, SymbolicLight uses 2,848 mJ/token versus 905 mJ/token for GPT-2 201M.
25 pages, 4 figures, 24 tables. Revised preprint: quality-gap framing, token-weighted versus unweighted domain PPL, and tightened architecture claims. Code and checkpoints: this https URL
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2605.21333 [cs.CL]
(or arXiv:2605.21333v2 [cs.CL] for this version)
Text integral sub licența deschisă a publicației · 234 cuvinte
Subiecte conexe
News
Quantum Computing Report · 25 iul. 2026
SymbolicLight V2: Hybrid Neuromorphic Architecture and Sparse Execution for Low-Energy Language Inference
arXiv · 10 sept. 2026
Article: Governing AI in the Cloud: A Practical Guide for Architects
InfoQ · 15 iun. 2026
Amazon Research Awards recipients announced
Amazon Science · 27 mai 2026