analiza aiqbench
SAUCE introduces a reference-free evaluation pipeline for conversational speech summarization that assesses accuracy through fact-based entailment rather than relying on black-box LLMs or dense embeddings. By mapping scores to discrete, verifiable facts, the method offers inherent explainability that allows developers to pinpoint specific hallucinations or omissions caused by transcription and translation errors. This approach provides a transparent tool for profiling LLM behaviors in noisy, real-world conditions where traditional metrics often fail.
SAUCE: Summary Analysis Using Conversation Entailment
arXiv:2611.06812v1 Announce Type: new Abstract: With the growing need for evaluating Large Language Models (LLMs) and their applications to speech, challenges persist in summarizing and evaluating conversations that lack a clear end goal. We introduce SAUCE -- a reference-free, fact-based evaluation pipeline for cross-lingual conversational speech summarization. It measures the accuracy and the fact coverage of a summary through the entailment between conversation and text. We compare SAUCE against several popular summarization metrics and demonstrate the effectiveness of capturing information loss due to transcription and translation error and identifying broken summaries. Crucially, unlike black-box LLM evaluators or dense embedding metrics, SAUCE is inherently explainable: it maps summary scores to discrete, verifiable facts, allowing users to pinpoint exact hallucinations or omissions. We illustrate how this interpretability helps developers systematically profile LLM behaviors and gives end-users an actionable tool to verify summary accuracy in noisy, real-world conditions. Preliminary investigations show SAUCE strongly align with human judgment.
Abstract:With the growing need for evaluating Large Language Models (LLMs) and their applications to speech, challenges persist in summarizing and evaluating conversations that lack a clear end goal. We introduce SAUCE -- a reference-free, fact-based evaluation pipeline for cross-lingual conversational speech summarization. It measures the accuracy and the fact coverage of a summary through the entailment between conversation and text. We compare SAUCE against several popular summarization metrics and demonstrate the effectiveness of capturing information loss due to transcription and translation error and identifying broken summaries. Crucially, unlike black-box LLM evaluators or dense embedding metrics, SAUCE is inherently explainable: it maps summary scores to discrete, verifiable facts, allowing users to pinpoint exact hallucinations or omissions. We illustrate how this interpretability helps developers systematically profile LLM behaviors and gives end-users an actionable tool to verify summary accuracy in noisy, real-world conditions. Preliminary investigations show SAUCE strongly align with human judgment.
Accepted and published to the ACL GEM workshop. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM), pages 364-377, San Diego, California, USA. Association for Computational Linguistics
Computation and Language (cs.CL); Audio and Speech Processing (eess.AS)
arXiv:2611.06812 [cs.CL]
(or arXiv:2611.06812v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2611.06812 arXiv-issued DOI via DataCite (pending registration)
https://doi.org/10.18653/v1/2026.gem-main.34 DOI(s) linking to related resources
Text integral sub licența deschisă a publicației · 208 cuvinte
Subiecte conexe
Anthropic reasoning without duplicates is just standard Bayesian updating
LessWrong · 30 iul. 2026
Culture Change for Successful Teams
Slack Blog · 17 sept. 2026
Do Diagrams Help Large Language Models Reason? Evidence from Syllogistic Reasoning
arXiv · 28 iul. 2026
What Is Contextual AI, and Why Does Context Matter for Today’s Tools?
Slack Blog · 16 oct. 2026