analiza aiqbench
This research introduces Flip-Flop Consistency ($F^2C$), an unsupervised training technique designed to mitigate LLM inconsistency when prompts are rephrased. By leveraging majority voting across prompt variations and representation alignment, the method significantly boosts agreement scores and reduces performance variance without requiring labeled data for every format. The findings demonstrate that enforcing consensus during training effectively generalizes to out-of-domain tasks, offering a practical path toward more robust language models.
Flip-Flop Consistency: Unsupervised Training for Robustness to Prompt Perturbations in LLMs
arXiv:2510.14242v2 Announce Type: replace Abstract: Large Language Models (LLMs) often produce inconsistent answers when faced with different phrasings of the same prompt. In this paper, we propose Flip-Flop Consistency ($F^2C$), an unsupervised training method that improves robustness to such perturbations. $F^2C$ is composed of two key components. The first, Consensus Cross-Entropy (CCE), uses a majority vote across prompt variations to create a hard pseudo-label. The second is a representation alignment loss that pulls lower-confidence and non-majority predictors toward the consensus established by high-confidence, majority-voting variations. We evaluate our method on 11 datasets spanning four NLP tasks, with 4-15 prompt variations per dataset. On average, $F^2C$ raises observed agreement by 11.62%, improves mean $F_1$ by 8.94%, and reduces performance variance across formats by 3.29%. In out-of-domain evaluations, $F^2C$ generalizes effectively, increasing $\overline{F_1}$ and agreement while decreasing variance across most source-target pairs. Finally, when trained on only a subset of prompt perturbations and evaluated on held-out formats, $F^2C$ consistently improves both performance and agreement while reducing variance. These findings highlight $F^2C$ as an effective unsupervised method for enhancing LLM consistency, performance, and generalization under prompt perturbations. Code is available at https://github.com/ParsaHejabi/Flip-Flop-Consistency-Unsupervised-Training-for-Robustness-to-Prompt-Perturbations-in-LLMs.
Abstract:Large Language Models (LLMs) often produce inconsistent answers when faced with different phrasings of the same prompt. In this paper, we propose Flip-Flop Consistency ($F^2C$), an unsupervised training method that improves robustness to such perturbations. $F^2C$ is composed of two key components. The first, Consensus Cross-Entropy (CCE), uses a majority vote across prompt variations to create a hard pseudo-label. The second is a representation alignment loss that pulls lower-confidence and non-majority predictors toward the consensus established by high-confidence, majority-voting variations. We evaluate our method on 11 datasets spanning four NLP tasks, with 4-15 prompt variations per dataset. On average, $F^2C$ raises observed agreement by 11.62%, improves mean $F_1$ by 8.94%, and reduces performance variance across formats by 3.29%. In out-of-domain evaluations, $F^2C$ generalizes effectively, increasing $\overline{F_1}$ and agreement while decreasing variance across most source-target pairs. Finally, when trained on only a subset of prompt perturbations and evaluated on held-out formats, $F^2C$ consistently improves both performance and agreement while reducing variance. These findings highlight $F^2C$ as an effective unsupervised method for enhancing LLM consistency, performance, and generalization under prompt perturbations. Code is available at this https URL.
17 pages, 7 figures, 7 tables, and 1 algorithm. ACL 2026. Revised after peer review; added Llama-3.2-3B-Instruct experiments, majority pseudo-label quality analysis, representation alignment analysis, and expanded implementation details
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2510.14242 [cs.CL]
(or arXiv:2510.14242v2 [cs.CL] for this version)
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1571-1587 (2026)
https://doi.org/10.18653/v1/2026.acl-long.71 DOI(s) linking to related resources
Text integral sub licența deschisă a publicației · 255 cuvinte
Subiecte conexe
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · 8 oct. 2026
PIRAMID: Progress and Plans
LessWrong · 27 iul. 2026
Grounding latent algorithm routing in transformer reasoning
arXiv · 28 iul. 2026
Informe de riesgos de la IA de frontera (febrero–marzo de 2026)
METR · 19 mai 2026