aiqbench analysis
This preprint stress-tests a frozen CNN used to screen Breakthrough Listen radio data, using 15,000 injected signals across ten morphologies and five amplitudes. Recall drops as signals get brighter, even for the needle morphology the detector was trained on, and one interference class yields a 0.627 false positive rate. The authors also show that reading only the ON panels leaves the detector unable to tell target-star signals from terrestrial interference, which makes in-distribution validation a weak guarantee. The results come from one model and one dataset, so they are a warning about validation practice more than a general verdict on ML in SETI.
Missed Detections and False Alarms in a CNN Technosignature Detector on Breakthrough Listen Data
arXiv:2610.22050v1 Announce Type: cross Abstract: Radio technosignature searches rely on machine learning detectors to shrink the candidate pool before a human looks at it. They are validated on held-out data from the training distribution, which excludes the cases where they are most likely to fail: a technosignature has unknown morphology, and terrestrial interference is unbounded in variety. I trained a standard competition detector once, froze it, and probed it with 15,000 injections across ten morphologies and five amplitudes, placed into real Green Bank backgrounds. Two directions were tested: genuine signals with morphologies other than a straight narrowband line, and interference resembling the training data. Recall did not follow the predicted ordering, and it fell as signals brightened, even for a needle of the kind the detector was trained on (from 0.590 to 0.090). One interference morphology reached a false positive rate of 0.627, comparable to the detector's own baseline recall of 0.5525 on genuine signals. Gradient-weighted Class Activation Mapping (Grad-CAM) showed attention along the injected path in one missed family by a pre-specified test, and visually in a second. A pre-specified, time-averaged ON-OFF filter removed false alarms and genuine detections at the same rate. The detector reads only the ON panels. Since the only evidence distinguishing a genuine candidate from terrestrial interference lies in the discarded OFF panels, it cannot tell if a signal originates from the target star; it can only tell if it resembles the training data. This explains the false alarm directly; the missed signals reflect what the detector learned to recognise, which reading the OFF panels would not by itself change.
Abstract:Radio technosignature searches rely on machine learning detectors to shrink the candidate pool before a human looks at it. They are validated on held-out data from the training distribution, which excludes the cases where they are most likely to fail: a technosignature has unknown morphology, and terrestrial interference is unbounded in variety. I trained a standard competition detector once, froze it, and probed it with 15,000 injections across ten morphologies and five amplitudes, placed into real Green Bank backgrounds. Two directions were tested: genuine signals with morphologies other than a straight narrowband line, and interference resembling the training data. Recall did not follow the predicted ordering, and it fell as signals brightened, even for a needle of the kind the detector was trained on (from 0.590 to 0.090). One interference morphology reached a false positive rate of 0.627, comparable to the detector's own baseline recall of 0.5525 on genuine signals. Gradient-weighted Class Activation Mapping (Grad-CAM) showed attention along the injected path in one missed family by a pre-specified test, and visually in a second. A pre-specified, time-averaged ON-OFF filter removed false alarms and genuine detections at the same rate. The detector reads only the ON panels. Since the only evidence distinguishing a genuine candidate from terrestrial interference lies in the discarded OFF panels, it cannot tell if a signal originates from the target star; it can only tell if it resembles the training data. This explains the false alarm directly; the missed signals reflect what the detector learned to recognise, which reading the OFF panels would not by itself change.
17 pages, 14 figures. Code and data: this https URL
Instrumentation and Methods for Astrophysics (astro-ph.IM); Machine Learning (cs.LG); Popular Physics (physics.pop-ph)
arXiv:2610.22050 [astro-ph.IM]
(or arXiv:2610.22050v1 [astro-ph.IM] for this version)
https://doi.org/10.48550/arXiv.2610.22050 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 373 words
Related coverage
2026.10: You are here 📍
Home Assistant · Oct 7, 2026
Learning to Detect Shifts in Unstructured Data Streams via Monitoring-by-Betting
arXiv · Oct 15, 2026
Online Change-point Detection for Cooperative Multi-Agent Reinforcement Learning
arXiv · Sep 7, 2026
The Blind Spot Paradox: When Adaptive Classifiers Defeat Drift Detectors
arXiv · Oct 6, 2026