aiqbench analysis
This study exposes a critical gap in current LLM evaluation by demonstrating that high accuracy scores do not guarantee answer stability when models face coherent counter-arguments. The researchers introduce the MaxFlip benchmark, which aggregates adversarial challenges across multiple models to significantly increase flip rates and better reveal underlying reasoning fragility. For developers and evaluators, this protocol offers a necessary new metric to assess robustness beyond simple correctness on standard datasets.
Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMs
arXiv:2606.16011v2 Announce Type: replace Abstract: Standard accuracy benchmarks evaluate whether large language models (LLMs) reach correct answers. However, they do not test whether models maintain that answer when challenged by a plausible counter-argument. We introduce a controlled protocol for evaluating answer stability: after a model answers a multiple-choice question correctly, we challenge the model's answer with a coherent argument for an incorrect option and measure whether the model flips. The setup a) isolates argumentative content from overt social pressure and b) varies argument length, self-attribution, and cross-model source. Across seven frontier models and 57 MMLU subjects, flip rates range from 17.5% to 97.3%, revealing large differences in stability that are not captured by accuracy metrics alone. We find that self-attribution consistently increases flip rates (mean 7.1pp, up to 18.7pp). Furthermore, pooling wrong-answer arguments across models and selecting the most effective one per question yields stronger adversarial challenges than relying on any single source model. From this cross-model pool, we construct MaxFlip, a curated benchmark that amplifies answer flips by up to 23.6pp over self-generated challenges. We release the protocol, challenge records, and MaxFlip to support stability evaluation alongside standard accuracy benchmarks. Materials are available at https://github.com/nafisenik/WhoFlips, https://hf.co/datasets/nafisehNik/WhoFlips.
Title:Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMs
Abstract:Standard accuracy benchmarks evaluate whether large language models (LLMs) reach correct answers. However, they do not test whether models maintain that answer when challenged by a plausible counter-argument. We introduce a controlled protocol for evaluating answer stability: after a model answers a multiple-choice question correctly, we challenge the model's answer with a coherent argument for an incorrect option and measure whether the model flips. The setup a) isolates argumentative content from overt social pressure and b) varies argument length, self-attribution, and cross-model source. Across seven frontier models and 57 MMLU subjects, flip rates range from 17.5% to 97.3%, revealing large differences in stability that are not captured by accuracy metrics alone. We find that self-attribution consistently increases flip rates (mean 7.1pp, up to 18.7pp). Furthermore, pooling wrong-answer arguments across models and selecting the most effective one per question yields stronger adversarial challenges than relying on any single source model. From this cross-model pool, we construct MaxFlip, a curated benchmark that amplifies answer flips by up to 23.6pp over self-generated challenges. We release the protocol, challenge records, and MaxFlip to support stability evaluation alongside standard accuracy benchmarks. Materials are available at this https URL, this https URL.
Computation and Language (cs.CL)
arXiv:2606.16011 [cs.CL]
(or arXiv:2606.16011v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.16011 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 300 words
Related coverage
Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMs
arXiv · Jun 17, 2026
The current bottleneck is political will, not research
AI Alignment Forum · Jul 11, 2026
The Entangled Dimensions of Decision Theory
LessWrong · Jul 30, 2026
Against Export Controls (and China Threat Models)
LessWrong · Sep 24, 2026