Quantum Attention by Overlap Interference: Predicting Classical and Many-Body Quantum Sequences

Abstract:We propose a variational quantum implementation of self-attention (QSA)-the core operation in transformers and large language models-which predicts future elements of a sequence by forming overlap-weighted combinations of past data. At variance with previous approaches, our QSA realizes the required nonlinearity through interference of state overlaps and a degree-$k$ polynomial kernel, and estimates a loss based on Rényi-$1/2$ entropic functionals via two observables' expectation values, avoiding the decoding of amplitude-encoded predictions into classical probabilities. QSA also accommodates a constrained, trainable data-embedding tying state overlaps to data-level similarities. Its dominant end-to-end training complexity scales as $O\left(\mu^{-1}k^2Td\right)$, versus $O\left(T d^{k+1}\right)$ of the fairest classical comparison, with $\mu$ a training signal; we show numerically that this allows a complexity advantage in the regime where sequence length $T$ dominates the embedding size $d$. In simulations, our QSA-based quantum transformer learns sequence prediction on classical data and on many-body transverse-field Ising trajectories-establishing trainable attention as a practical primitive for quantum dynamical modeling.

Quantum Physics (quant-ph); Computation and Language (cs.CL); Machine Learning (cs.LG)

arXiv:2602.06699 [quant-ph]

(or arXiv:2602.06699v2 [quant-ph] for this version)

https://doi.org/10.48550/arXiv.2602.06699 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.