When do cheap embeddings beat protein language models? A theoretically-grounded hashing sketch for biological sequence classification

Important: e-prints posted on arXiv are not peer-reviewed by arXiv; they should not be relied upon without context to guide clinical practice or health-related behavior and should not be reported in news media as established information without consulting multiple experts in the field.

Title:When do cheap embeddings beat protein language models? A theoretically-grounded hashing sketch for biological sequence classification

Abstract:\textbf{Motivation:} Pre-trained protein language models (PLMs) such as ESM-2 have become the default representation for biological sequence tasks, but they are computationally heavy and require GPUs both for embedding and for fine-tuning. Whether they are actually necessary for sequence \emph{classification}, as opposed to structure prediction, is rarely tested against strong, principled, lightweight alternatives. This question has direct practical stakes for large-scale genomic surveillance, where embedding millions of sequences on commodity hardware is a recurring bottleneck.\\ \textbf{Results:} We introduce Murmur2Vec, an alignment-free, training-free embedding that aggregates $k$-mer counts into a small hash table via the deterministic MurmurHash function, and we cast it as a randomized sketch of the classical $k$-mer spectrum kernel. We provide a complete theoretical treatment: closed-form bias/variance of the inner product, an unbiased signed variant with a Johnson--Lindenstrauss-type concentration bound, an excess-risk bound for downstream linear classifiers that makes the bias--variance trade-off in the hash-table size explicit, and an implicit-regularization mechanism by which collisions damage frequent non-discriminative $k$-mers more than rare lineage-defining ones. Across four classification tasks, SARS-CoV-2 spike lineage (22 classes), HIV-1 Env subtype (8 classes), and two protein-family benchmarks (8 and 6 classes), Murmur2Vec matches a LoRA-fine-tuned 650M-parameter ESM-2 model on the two tasks for which LoRA fine-tuning was run to convergence (SARS-CoV-2 and HIV-1) and ties frozen ESM-2 on the two protein-family tasks, and it \emph{outperforms} the fine-tuned model on the hardest task (SARS-CoV-2 lineage: $0.854$ vs.\ $0.807$ accuracy; macro-F1 $0.684$ vs.\ $0.401$).

Machine Learning (cs.LG); Genomics (q-bio.GN)

arXiv:2512.10147 [cs.LG]

(or arXiv:2512.10147v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2512.10147 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.