aiqbench analysis
This research exposes a critical gap in LLM evaluation by demonstrating that standard accuracy metrics fail to capture answer instability when models are challenged with plausible counter-arguments. The study reveals significant variability across frontier models, showing how factors like self-attribution and cross-model adversarial inputs can drastically increase the likelihood of an incorrect "flip." By releasing the MaxFlip challenge set and a new protocol, this work provides essential tools for assessing robustness beyond simple correctness rates.
Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMs
arXiv:2606.16011v1 Announce Type: new Abstract: Standard accuracy benchmarks are designed to test how closely large language models (LLMs) approach correct answers, but are not suitable for testing whether LLMs stick with a correct answer when that answer is challenged by a plausible counter-argument. We introduce a controlled protocol for evaluating answer stability: after a model answers a multiple-choice question correctly, we challenge the model's answer with a coherent argument for an incorrect option and measure whether the model flips. The setup a) isolates argumentative content from overt social pressure and b) varies argument length, self-attribution, and cross-model source. Across seven frontier models and 57 MMLU subjects, flip rates range from 17.5% to 97.3%, revealing large differences in stability that are not captured by accuracy metrics alone. We find that self-attribution consistently increases flip rates (mean +7.1pp, up to +18.7pp). Also, pooling wrong-answer arguments across models and selecting the most effective one per question yields stronger adversarial challenges than relying on any single source model. We further construct MaxFlip, a curated challenge set that amplifies flips by up to +23.6pp over standard self-generated challenges. We release the protocol, challenge records, and MaxFlip to support stability evaluation alongside standard accuracy benchmarks. Materials are available at https://github.com/nafisenik/WhoFlips and https://hf.co/datasets/nafisehNik/WhoFlips.
Title:Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMs
Abstract:Standard accuracy benchmarks are designed to test how closely large language models (LLMs) approach correct answers, but are not suitable for testing whether LLMs stick with a correct answer when that answer is challenged by a plausible counter-argument. We introduce a controlled protocol for evaluating answer stability: after a model answers a multiple-choice question correctly, we challenge the model's answer with a coherent argument for an incorrect option and measure whether the model flips. The setup a) isolates argumentative content from overt social pressure and b) varies argument length, self-attribution, and cross-model source. Across seven frontier models and 57 MMLU subjects, flip rates range from 17.5% to 97.3%, revealing large differences in stability that are not captured by accuracy metrics alone. We find that self-attribution consistently increases flip rates (mean +7.1pp, up to +18.7pp). Also, pooling wrong-answer arguments across models and selecting the most effective one per question yields stronger adversarial challenges than relying on any single source model. We further construct MaxFlip, a curated challenge set that amplifies flips by up to +23.6pp over standard self-generated challenges. We release the protocol, challenge records, and MaxFlip to support stability evaluation alongside standard accuracy benchmarks. Materials are available at this https URL and this https URL.
Computation and Language (cs.CL)
arXiv:2606.16011 [cs.CL]
(or arXiv:2606.16011v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.16011 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 311 words
Related coverage
Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMs
arXiv · Sep 1, 2026
The current bottleneck is political will, not research
AI Alignment Forum · Jul 11, 2026
The Entangled Dimensions of Decision Theory
LessWrong · Jul 30, 2026
Against Export Controls (and China Threat Models)
LessWrong · Sep 24, 2026