Pre-Whitening and BCJR Posterior Distillation for Bi-LSTM Detection in Faster-than-Nyquist Signaling

Abstract:Recurrent detectors such as bidirectional long short-term memory (Bi-LSTM) networks are low-complexity alternatives to the optimal Bahl-Cocke-Jelinek-Raviv (BCJR) detector for faster-than-Nyquist (FTN) signaling. Motivated by convolutional detectors that build the intersymbol interference (ISI) structure into their architecture, we ask whether processing nested ISI windows in separate recurrent branches improves the bit error rate (BER) of a Bi-LSTM. Across roughly 260 controlled trainings it does not: at a matched parameter budget and a matched readout, the multi-window architecture never significantly beats a plain Bi-LSTM. Nested windowing is an invertible rearrangement that adds no information, extra branches only add bottlenecks, and a distillation diagnostic shows the network is already near optimal for its window. The limitation is therefore the observation model, not the architecture. Keeping the architecture fixed, we pre-whiten the input, restoring the conditional independence that colored matched-filter noise violates, and distill the BCJR soft posterior into the network. With 3.4% more parameters this reaches 1.05 times the BCJR BER at a compression factor of 0.8 and 1.89 times at 0.7, improving to 1.47 times when the whitened window is widened. The 23.7% BER reduction at 0.8 requires an ill-conditioned ISI matrix but is not monotone in the conditioning, and it holds across five independent noise realizations and a symbol-level McNemar test.

Accepted and presented at IDAP 2026. 5 pages, 13 figures

Signal Processing (eess.SP); Machine Learning (cs.LG)

arXiv:2609.07762 [eess.SP]

(or arXiv:2609.07762v1 [eess.SP] for this version)

https://doi.org/10.48550/arXiv.2609.07762 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.