aiqbench analysis
This research addresses the reusability of DeepSeek's Engram module by replacing XOR-based hashing with polynomial methods to create a joint embedding space compatible across different tokenizers. By treating N-grams as byte sequence samplers rather than disjoint spaces, the approach decouples memory modules from specific tokenizer architectures without sacrificing performance. This advancement is significant for building modular AI systems where reasoning capabilities can be transferred between models using diverse vocabulary encodings.
Tokenizer-Agnostic Engram Module
arXiv:2607.29065v1 Announce Type: new Abstract: Deepseek's Engram, a conditional memory module, was introduced to trade-off storage versus reasoning in large language models. However, the module relies on token-level $N$-gram hashing for Engram embedding lookup, introducing a tight coupling to the tokenizer used: a model with a different tokenizer would have to train its own Engram embeddings from scratch. To improve the reusability of Engram embeddings, we propose a change to the hashing routine, enabling compatibility between Engram models using different tokenizers. Instead of modelling disjoint $N$-gram spaces, we treat $N$-gram as a method to sample potentially useful byte sequences, from all possible byte sequences across tokens. We replace the XOR-based hashing with the general polynomial hashing with a joint embedding space across $N$. This work investigates the possible trade-offs and shows that this simple substitution produces comparable performance and achieves tokenizer-agnosticism: hash equivalence for byte-equivalent token sequences.
Abstract:Deepseek's Engram, a conditional memory module, was introduced to trade-off storage versus reasoning in large language models. However, the module relies on token-level $N$-gram hashing for Engram embedding lookup, introducing a tight coupling to the tokenizer used: a model with a different tokenizer would have to train its own Engram embeddings from scratch. To improve the reusability of Engram embeddings, we propose a change to the hashing routine, enabling compatibility between Engram models using different tokenizers. Instead of modelling disjoint $N$-gram spaces, we treat $N$-gram as a method to sample potentially useful byte sequences, from all possible byte sequences across tokens. We replace the XOR-based hashing with the general polynomial hashing with a joint embedding space across $N$. This work investigates the possible trade-offs and shows that this simple substitution produces comparable performance and achieves tokenizer-agnosticism: hash equivalence for byte-equivalent token sequences.
Computation and Language (cs.CL)
arXiv:2607.29065 [cs.CL]
(or arXiv:2607.29065v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2607.29065 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 237 words
Related coverage
Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs
arXiv · Jun 9, 2026
CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models
arXiv · Aug 5, 2026
TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference
arXiv · Sep 24, 2026
Model-agnostic PII detection with LLMs
AWS · Sep 10, 2026