analiza aiqbench
This paper addresses a critical bottleneck in Large Language Model evaluation by introducing ConQual-Audit, a framework designed to eliminate false positives and negatives caused by overly restrictive or incomplete constraint sets during personalization tasks. By automatically pruning unnecessary constraints and recovering missing ones before verification, the method significantly improves evaluation accuracy across standard benchmarks like PrefEval and LaMP while maintaining low computational overhead. The work advances the field of machine learning by providing a more robust methodology for assessing how well LLMs adhere to specific user preferences without introducing artificial limitations.
Neither Over- nor Under-Constrained: Evaluating LLM Personalization under a Constraint Qualification Audit
arXiv:2610.25676v1 Announce Type: new Abstract: Accurate evaluation of LLM personalization is hindered by current evaluation paradigms that over- or under-constrain the response space, yielding frequent false rejections and false acceptances. To reduce these false rates, we introduce ConQual-Audit, an audit-based framework that evaluates LLM personalization through a three-stage process: (1) extracting semantic constraints from user profiles and task-grounded reference; (2) qualifying constraints by pruning unnecessary constraints and recovering missing constraints; and (3) verifying candidate responses against the audited constraints via natural language inference. Across PrefEval and LaMP, ConQual-Audit outperforms nine baselines, attaining top AUCs (0.88 and 0.90) and the lowest false rate (0.21) on the leanest constraint sets at an amortizable audit cost. Crucially, this constraint audit is portable: integrating it into ExPerT and PREF helps revise smaller, higher-quality constraint sets to reduce their evaluation false rates.
Abstract:Accurate evaluation of LLM personalization is hindered by current evaluation paradigms that over- or under-constrain the response space, yielding frequent false rejections and false acceptances. To reduce these false rates, we introduce ConQual-Audit, an audit-based framework that evaluates LLM personalization through a three-stage process: (1) extracting semantic constraints from user profiles and task-grounded reference; (2) qualifying constraints by pruning unnecessary constraints and recovering missing constraints; and (3) verifying candidate responses against the audited constraints via natural language inference. Across PrefEval and LaMP, ConQual-Audit outperforms nine baselines, attaining top AUCs (0.88 and 0.90) and the lowest false rate (0.21) on the leanest constraint sets at an amortizable audit cost. Crucially, this constraint audit is portable: integrating it into ExPerT and PREF helps revise smaller, higher-quality constraint sets to reduce their evaluation false rates.
Artificial Intelligence (cs.AI)
arXiv:2610.25676 [cs.AI]
(or arXiv:2610.25676v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.25676 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 224 cuvinte