aiqbench analysis
The paper introduces a benchmark for detecting phoneme-level errors in speech, built from child recordings, scripted actor recordings, and synthetic speech. Its main finding is that 13 current speech models struggle to detect these errors reliably, and they often flag correct pronunciations as errors, so precision is low. Speech-language models fine-tuned on synthetic data do better than commercial and open-source baselines on word-level detection, but a sim-to-real gap remains. For readers, the practical takeaway is that clinical and educational speech tools built on these models need stronger validation before deployment.
PEDBench: Can Speech Models Detect Phonetic Errors?
arXiv:2610.20830v1 Announce Type: new Abstract: Phonetic error detection (PED) aims to identify phoneme-level substitutions, deletions, and insertions and is essential for educational and clinical speech assessment and intervention. Despite recent advances in speech models for phoneme recognition, their ability to reliably detect fine-grained phonetic errors remains unclear because of a lack of high-quality evaluation data. To address this gap, we introduce PEDBench, the first PED benchmark spanning a continuum from natural to controlled speech. It comprises real-world recordings of spoken errors made by children (N=334), scripted error productions by human actors (N=1,188), and synthetic error productions generated using phone-conditioned text-to-speech (N=1,083). All phonetic annotations were transcribed and verified by two phonetically trained annotators. Our evaluation of 13 state-of-the-art speech models shows that they fail to reliably detect phonetic errors, even under controlled conditions with acted and synthetic speech, where acoustic variability is reduced. The models also exhibit low precision, frequently misclassifying correctly pronounced phonemes as errors. Our speech-language models, fine-tuned on synthetic data, outperform commercial and open-source baselines on word-level PED, although a sim-to-real domain gap remains. We plan to publicly release our code and recipes, along with our acted and synthetic datasets, to support the development of more accurate PED systems.
Title:PEDBench: Can Speech Models Detect Phonetic Errors?
Abstract:Phonetic error detection (PED) aims to identify phoneme-level substitutions, deletions, and insertions and is essential for educational and clinical speech assessment and intervention. Despite recent advances in speech models for phoneme recognition, their ability to reliably detect fine-grained phonetic errors remains unclear because of a lack of high-quality evaluation data. To address this gap, we introduce PEDBench, the first PED benchmark spanning a continuum from natural to controlled speech. It comprises real-world recordings of spoken errors made by children (N=334), scripted error productions by human actors (N=1,188), and synthetic error productions generated using phone-conditioned text-to-speech (N=1,083). All phonetic annotations were transcribed and verified by two phonetically trained annotators. Our evaluation of 13 state-of-the-art speech models shows that they fail to reliably detect phonetic errors, even under controlled conditions with acted and synthetic speech, where acoustic variability is reduced. The models also exhibit low precision, frequently misclassifying correctly pronounced phonemes as errors. Our speech-language models, fine-tuned on synthetic data, outperform commercial and open-source baselines on word-level PED, although a sim-to-real domain gap remains. We plan to publicly release our code and recipes, along with our acted and synthetic datasets, to support the development of more accurate PED systems.
Computation and Language (cs.CL); Sound (cs.SD)
arXiv:2610.20830 [cs.CL]
(or arXiv:2610.20830v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.20830 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 301 words
Related coverage
The Open ASR Leaderboard Adds Its First Global South Language
Hugging Face · Aug 28, 2026
Evaluate Clinical ASR Models Faster with Agent Skills and NVIDIA Nemotron Speech
NVIDIA · Jun 9, 2026
FuseAlign: Forced Alignment in the Wild
arXiv · Sep 30, 2026
Automatic Pronunciation Error Detection and Correction of the Holy Quran's Learners Using Deep Learning
arXiv · Sep 1, 2026