aiqbench analysis
This research introduces SMART, a novel framework for evaluating automatic speech recognition (ASR) systems by mapping lexical errors to their specific semantic consequences rather than relying on traditional word error rates. By utilizing contextual rubrics, the method achieves high accuracy in identifying meaning-altering mistakes and provides a robust metric for targeted model correction that is less sensitive to the underlying evaluation LLM. This advancement addresses a critical gap in ASR quality assurance, enabling developers to prioritize fixes that genuinely impact user understanding over those with negligible semantic cost.
SMART: Semantic Segment Mapping and Assessment of Recognition Transcripts with Contextual Rubrics
arXiv:2610.27820v1 Announce Type: new Abstract: Word error rate (WER) assigns equal cost to lexical edits despite their different semantic consequences, while overall semantic scores cannot identify which individual errors alter the intended meaning. We formulate ASR semantic evaluation as context-aware, segment-level consequence assessment and propose Semantic Segment Mapping and Assessment of Recognition Transcripts with Contextual Rubrics (SMART), a framework for traceable semantic evaluation and targeted ASR correction. SMART maps reference-hypothesis discrepancies into coherent segments and applies a structured contextual rubric to assign explicit consequence labels. These labels support Semantic Impact Error Rate (SIER), which quantifies consequential errors at utterance, chunk, and lexical levels. Against majority-vote human labels, SMART achieves 0.843 F1 versus 0.745 for direct LLM judgment. Evaluation across multiple LLMs further demonstrates that SMART is relatively insensitive to the choice of evaluation model, while MER and SIER reveal different aspects of ASR quality. A separate target-reference-free rubric guides correction using the target hypothesis and surrounding ASR hypotheses, outperforming a matched w/o Rubric baseline on both datasets, with further absolute reductions of 0.85-0.98 percentage points in MER and 1.18-1.65 percentage points in SIER_lex.
Abstract:Word error rate (WER) assigns equal cost to lexical edits despite their different semantic consequences, while overall semantic scores cannot identify which individual errors alter the intended meaning. We formulate ASR semantic evaluation as context-aware, segment-level consequence assessment and propose Semantic Segment Mapping and Assessment of Recognition Transcripts with Contextual Rubrics (SMART), a framework for traceable semantic evaluation and targeted ASR correction. SMART maps reference-hypothesis discrepancies into coherent segments and applies a structured contextual rubric to assign explicit consequence labels. These labels support Semantic Impact Error Rate (SIER), which quantifies consequential errors at utterance, chunk, and lexical levels. Against majority-vote human labels, SMART achieves 0.843 F1 versus 0.745 for direct LLM judgment. Evaluation across multiple LLMs further demonstrates that SMART is relatively insensitive to the choice of evaluation model, while MER and SIER reveal different aspects of ASR quality. A separate target-reference-free rubric guides correction using the target hypothesis and surrounding ASR hypotheses, outperforming a matched w/o Rubric baseline on both datasets, with further absolute reductions of 0.85-0.98 percentage points in MER and 1.18-1.65 percentage points in SIER_lex.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Audio and Speech Processing (eess.AS)
arXiv:2610.27820 [cs.CL]
(or arXiv:2610.27820v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.27820 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 281 words
Related coverage
From Transcription to Reformulation: Defining and Benchmarking Grounded Generative Speech Recognition
arXiv · Nov 2, 2026
Autoresearch with Coding Agents: Generalizers and Metric-Maximizers on Quran Recitation Data
arXiv · Jul 21, 2026
Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs
arXiv · Oct 16, 2026
Qwen-Audio-3.1-ASR Technical Report
arXiv · Oct 22, 2026