aiqbench analysis
This study introduces a rubric-guided SpeechLLM that jointly predicts multi-granular proficiency labels and generates natural-language rationales for L2 speech assessment. While the model achieves competitive performance on the SpeechOcean762 dataset, analysis reveals that while sentence-level rationales are plausible, faithfulness degrades significantly at the word and phoneme levels due to sparse references. This highlights a critical interpretability gap between high-level fluency judgments and fine-grained acoustic accuracy in current automated assessment frameworks.
A Finetuned SpeechLLM for Joint Multi-Granular L2 Assessment and Natural-Language Rationales
arXiv:2606.09470v1 Announce Type: new Abstract: Automated L2 speech assessment can assign proficiency labels, but often lacks interpretability. We propose a rubric-guided SpeechLLM for multi-aspect, multi-granular assessment, trained with a hybrid objective combining supervised fine-tuning and Bounded Direct Preference Optimization. The model jointly predicts ordinal labels at the sentence-level (accuracy, fluency, prosody), word/phoneme-level accuracy, and generates a natural-language rationale in the same response. On SpeechOcean762, our approach matches or outperforms single-granularity models while remaining competitive with prior approaches. We analyze rationale reliability along two axes: self-consistency with model predictions and alignment with ground-truth labels, using sentiment consistency (plausibility) and mention-based agreement (faithfulness). Rationales are plausible at the sentence level, but faithfulness degrades at the word/phoneme level: references are sparse and weakly aligned with token-level labels.
Abstract:Automated L2 speech assessment can assign proficiency labels, but often lacks interpretability. We propose a rubric-guided SpeechLLM for multi-aspect, multi-granular assessment, trained with a hybrid objective combining supervised fine-tuning and Bounded Direct Preference Optimization. The model jointly predicts ordinal labels at the sentence-level (accuracy, fluency, prosody), word/phoneme-level accuracy, and generates a natural-language rationale in the same response. On SpeechOcean762, our approach matches or outperforms single-granularity models while remaining competitive with prior approaches. We analyze rationale reliability along two axes: self-consistency with model predictions and alignment with ground-truth labels, using sentiment consistency (plausibility) and mention-based agreement (faithfulness). Rationales are plausible at the sentence level, but faithfulness degrades at the word/phoneme level: references are sparse and weakly aligned with token-level labels.
Accepted to Interspeech 2026. This publication is part of the project Responsible AI for Voice Diagnostics (RAIVD) with file number NGF.1607.22.013 of the research programme NGF AiNed Fellowship Grants, which is financed by the Dutch Research Council (NWO)
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2606.09470 [cs.CL]
(or arXiv:2606.09470v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.09470 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 179 words