Matterhorn: Masked Time-to-First-Spike Encoding by Reassigning the Silent State for Sparse and Energy-Efficient Spiking Transformers

Abstract:Spiking neural networks (SNNs) promise energy-efficient inference for large language models (LLMs), yet most reported savings rely on compute-operation counts that overlook data movement. Energy characterization of representative spiking transformers on a commercial 22-nm process shows that accumulation contributes less than 3% of total energy, while spike-triggered inter-core transfers and weight reads dominate the cost. This makes time-to-first-spike (TTFS) encoding a natural choice, as it limits each neuron to at most one spike. However, standard TTFS maps the silent state, an all-zero spike train that transmits no events, to the rarely occurring smallest value, while the most common activations still spike. This raises a simple question: why reserve the only cost-free codeword for a rare value? This choice inverts a basic principle of energy-aware coding, under which the zero-event codeword should represent the most common value, rather than a rare extreme. Thus, we introduce masked time-to-first-spike encoding (M-TTFS), which uses a temporal mask to reassign the silent state to the most common activation value, and a dead-zone extension that trades a controlled amount of information for greater sparsity. Built on M-TTFS with dead-zone radius $k{=}1$, our spiking transformer Matterhorn reaches an overall spike rate of 1.64% on GLUE at an average score of 84.64, exceeding the best prior spiking transformer by 1.42 percentage points while consuming 67% less energy, with consistent gains on spiking LLaMA models from 7B to 70B parameters. Together, these results show that under hardware-faithful accounting, the energy advantage of SNNs is not a given: it is earned by encodings that align spikes with the data distribution.

Machine Learning (cs.LG)

arXiv:2601.22876 [cs.LG]

(or arXiv:2601.22876v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2601.22876 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.