aiqbench analysis
This is a machine learning paper: NOEMA, a temporal convolutional decoder, is trained to match fMRI windows to multimodal speech embeddings, and the reported gains come from its architecture and representation choices. The headline result is retrieval accuracy (the correct segment in the top 10 of 2,000 candidates for over 42% of held-out queries), which is a useful benchmark for neural decoding but not a demonstration of reconstructing perceived speech. Readers should also note that the evaluation rests on two datasets and that results on fMRI's slow, smeared signal may not transfer to other recording modalities.
Decoding Naturalistic Speech from fMRI with Temporal Modeling and Multimodal Alignment
arXiv:2610.22049v1 Announce Type: cross Abstract: Reconstructing what a person perceives from their brain activity is a stringent test of whether neural recordings carry meaningful information. Naturalistic speech offers a rich testbed for this challenge: it unfolds over multiple timescales, but fMRI's noisy, temporally smeared measurements make this structure hard to recover. We introduce NOEMA, a temporal convolutional decoder that aligns windows of consecutive fMRI measurements with multimodal stimulus embeddings. These embeddings combine complementary text, audio, and speech representations of the same stimulus segment, enabling its retrieval directly from brain activity. On two naturalistic speech datasets, Cephalonauts One and LeBel, NOEMA retrieves the correct segment among 2,000 candidates within the top 10 for more than 42% of held-out queries. This substantially outperforms baseline decoders, an advantage that persists when the candidate pool is extended with thousands of distractors absent from training. Controlled ablations establish the contributions of temporal modeling and complementary stimulus representations.
Abstract:Reconstructing what a person perceives from their brain activity is a stringent test of whether neural recordings carry meaningful information. Naturalistic speech offers a rich testbed for this challenge: it unfolds over multiple timescales, but fMRI's noisy, temporally smeared measurements make this structure hard to recover. We introduce NOEMA, a temporal convolutional decoder that aligns windows of consecutive fMRI measurements with multimodal stimulus embeddings. These embeddings combine complementary text, audio, and speech representations of the same stimulus segment, enabling its retrieval directly from brain activity. On two naturalistic speech datasets, Cephalonauts One and LeBel, NOEMA retrieves the correct segment among 2,000 candidates within the top 10 for more than 42% of held-out queries. This substantially outperforms baseline decoders, an advantage that persists when the candidate pool is extended with thousands of distractors absent from training. Controlled ablations establish the contributions of temporal modeling and complementary stimulus representations.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.22049 [cs.LG]
(or arXiv:2610.22049v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.22049 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 244 words
Related coverage
From Neurons to Conversation: Speech Brain-Computer Interfaces
arXiv · Oct 1, 2026
Decoding Naturalistic Emotion Dynamics from the Brain: An LLM-Enhanced Regression Framework
arXiv · Jun 9, 2026
BAT-CLIP: Trimodal Alignment of Brain, Audio and Text
arXiv · Sep 28, 2026
Contextual Semantic Relevance Tracks fMRI BOLD Responses During Naturalistic Speech Comprehension
arXiv · Jul 20, 2026