aiqbench analysis
This paper introduces S$^3$F-Net, a novel architecture that enhances medical image classification by fusing spatial convolutional features with frequency-domain spectral representations. By leveraging the Convolution Theorem to process Fourier spectra directly, the model achieves significant accuracy gains while maintaining computational efficiency through lightweight summarizer networks. This approach offers a concrete advancement in machine learning methodologies for biomedical imaging, demonstrating how multi-modal feature fusion can outperform traditional spatial-only baselines.
S$^3$F-Net: A Multi-Modal Approach to Medical Image Classification via Spatial-Spectral Summarizer Fusion Network
arXiv:2509.23442v2 Announce Type: replace-cross Abstract: Convolutional Neural Networks have become a cornerstone of medical image analysis due to their proficiency in learning hierarchical spatial features. However, this focus on a single domain is inefficient at capturing global, holistic patterns and fails to explicitly model an image's frequency-domain characteristics. To address these challenges, we propose the Spatial-Spectral Summarizer Fusion Network (S$^3$F-Net), a dual-branch framework that learns from both spatial and spectral representations simultaneously. The S$^3$F-Net performs a fusion of a deep spatial CNN with our proposed shallow spectral encoder, SpectraNet. SpectraNet features the proposed SpectralFilter layer, which leverages the Convolution Theorem by applying a bank of learnable filters directly to an image's full Fourier spectrum via a computation-efficient element-wise multiplication. This allows the SpectralFilter layer to attain a global receptive field instantaneously, with its output being distilled by a lightweight summarizer network. We evaluate S$^3$F-Net across four medical imaging datasets spanning different modalities to validate its efficacy and generalizability. Our framework consistently and significantly outperforms its strong spatial-only baseline in all cases, with accuracy improvements of up to 5.13%. With a powerful Bilinear Fusion, S$^3$F-Net achieves a SOTA competitive accuracy of 98.76% on the BRISC2025 dataset. Concatenation Fusion performs better on the texture-dominant Chest X-Ray Pneumonia dataset, achieving 93.11% accuracy, surpassing many top-performing, much deeper models. Our explainability analysis also reveals that the S$^3$F-Net learns to dynamically adjust its reliance on each branch based on the input pathology. Our source code and pre-trained model weights are publicly available on GitHub at: https://github.com/Saiful185/S3F-Net.
Abstract:Convolutional Neural Networks have become a cornerstone of medical image analysis due to their proficiency in learning hierarchical spatial features. However, this focus on a single domain is inefficient at capturing global, holistic patterns and fails to explicitly model an image's frequency-domain characteristics. To address these challenges, we propose the Spatial-Spectral Summarizer Fusion Network (S$^3$F-Net), a dual-branch framework that learns from both spatial and spectral representations simultaneously. The S$^3$F-Net performs a fusion of a deep spatial CNN with our proposed shallow spectral encoder, SpectraNet. SpectraNet features the proposed SpectralFilter layer, which leverages the Convolution Theorem by applying a bank of learnable filters directly to an image's full Fourier spectrum via a computation-efficient element-wise multiplication. This allows the SpectralFilter layer to attain a global receptive field instantaneously, with its output being distilled by a lightweight summarizer network. We evaluate S$^3$F-Net across four medical imaging datasets spanning different modalities to validate its efficacy and generalizability. Our framework consistently and significantly outperforms its strong spatial-only baseline in all cases, with accuracy improvements of up to 5.13%. With a powerful Bilinear Fusion, S$^3$F-Net achieves a SOTA competitive accuracy of 98.76% on the BRISC2025 dataset. Concatenation Fusion performs better on the texture-dominant Chest X-Ray Pneumonia dataset, achieving 93.11% accuracy, surpassing many top-performing, much deeper models. Our explainability analysis also reveals that the S$^3$F-Net learns to dynamically adjust its reliance on each branch based on the input pathology. Our source code and pre-trained model weights are publicly available on GitHub at: this https URL.
Published in IEEE Journal of Biomedical and Health Informatics (JBHI). Find the published version at: this https URL
Image and Video Processing (eess.IV); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Signal Processing (eess.SP)
arXiv:2509.23442 [eess.IV]
(or arXiv:2509.23442v2 [eess.IV] for this version)
https://doi.org/10.1109/JBHI.2026.3682634 DOI(s) linking to related resources
From: Md. Saiful Bari Siddiqui [view email] [v1] Sat, 27 Sep 2025 18:18:39 UTC (1,186 KB) [v2] Sat, 5 Sep 2026 19:25:13 UTC (1,940 KB)
Full text under the publisher's open licence · 325 words
Related coverage
Introducing CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement
Microsoft Research · Aug 11, 2026
MWIR-4-Plastic: The Identification of Complex End-of-Life Industrial Plastic using Mid-wave Infrared Hyperspectral Imaging and Machine Learning
arXiv · Sep 1, 2026
Deep Learning for Longitudinal Medical Imaging: A Scoping Review
arXiv · Oct 8, 2026
Synthesize Realistic 3D Medical Images at Scale to Ship Pre‑Trained Models
NVIDIA · May 22, 2026