aiqbench analysis
StanceBench introduces a critical evaluation framework for audio-capable LLMs by quantifying their ability to detect interpersonal nuances like empathy and honesty in speech. The benchmark reveals significant challenges, such as high prompt order bias when assessing truthfulness and weak alignment with human judgment on attentiveness metrics. This work advances the field of multimodal AI by establishing standardized protocols for testing social intent inference beyond simple text processing.
StanceBench: A Benchmark for Audio LLM-Based Interpersonal Stance Evaluation from Speech
arXiv:2607.22658v1 Announce Type: new Abstract: Speech-to-speech dialogue models increasingly depend on prosody and interactional nuance to convey social intent, yet benchmarks for these cues remain limited. We introduce StanceBench, a benchmark for measuring interpersonal stance in conversational speech and evaluating audio-capable LLMs as automated judges. Using the Seamless Interaction corpus, StanceBench (1) specifies 9 stance dimensions via role-prompt poles, (2) standardizes single-speaker and interaction-based evaluations, and (3) reports LLM-as-a-judge robustness, bias, and stance inference. Across evaluated stances, empathy and politeness are the easiest. Warmth and assertiveness are moderately separable with positivity skew/asymmetry. Honesty is the hardest and shows high prompt order bias, consistent with needing cross-turn evidence. Attentiveness is separable but aligns weakly with humans. Interaction stances are more context-sensitive, with threshold gaps and high variance, especially conflict regulation.
Abstract:Speech-to-speech dialogue models increasingly depend on prosody and interactional nuance to convey social intent, yet benchmarks for these cues remain limited. We introduce StanceBench, a benchmark for measuring interpersonal stance in conversational speech and evaluating audio-capable LLMs as automated judges. Using the Seamless Interaction corpus, StanceBench (1) specifies 9 stance dimensions via role-prompt poles, (2) standardizes single-speaker and interaction-based evaluations, and (3) reports LLM-as-a-judge robustness, bias, and stance inference. Across evaluated stances, empathy and politeness are the easiest. Warmth and assertiveness are moderately separable with positivity skew/asymmetry. Honesty is the hardest and shows high prompt order bias, consistent with needing cross-turn evidence. Attentiveness is separable but aligns weakly with humans. Interaction stances are more context-sensitive, with threshold gaps and high variance, especially conflict regulation.
Artificial Intelligence (cs.AI); Sound (cs.SD); Audio and Speech Processing (eess.AS)
arXiv:2607.22658 [cs.AI]
(or arXiv:2607.22658v1 [cs.AI] for this version)
Full text under the publisher's open licence · 142 words
Related coverage
[AINews] High Taste Testers Ep. 1
Latent Space · Oct 17, 2026
A Candid Abacus AI Review: The All-in-One AI Platform for Professionals & Enterprises
KDnuggets · Sep 10, 2026
When Text Misleads: Inconsistent-Aware Reasoning for Audio-Grounded Dialogue
arXiv · Aug 28, 2026
Benchtalks #2: The future of coding benchmarks
Snorkel AI Blog · Jun 3, 2026