aiqbench analysis
This research exposes a critical vulnerability in the AI development lifecycle, demonstrating how the pressure to improve benchmark scores can be weaponized to implant backdoors into large language models like Qwen and Gemma. The study reveals that adversaries can simultaneously achieve genuine performance gains on standard tests while maintaining high success rates for malicious triggers, even when employing data decontamination techniques. These findings underscore the urgent need to decouple evaluation metrics from training incentives to prevent benchmarks from becoming vectors for systemic model compromise.
Adversarial Benchmarks and BenchDoor Attacks: Exploiting Benchmark Incentives for Backdoor Attacks
arXiv:2610.27533v1 Announce Type: cross Abstract: Machine learning benchmarks act as guideposts for model development and spur the creation of training datasets intended to improve benchmark scores. We introduce BenchDoor, a family of data poisoning attacks that exploits this incentive. Beyond preserving benign performance, BenchDoor requires benchmark gains that encourage adoption of data that also implants a backdoor. The adversary either manipulates a benchmark's training or test data, creating an adversarial benchmark, or supplies an independent training dataset while leaving the benchmark unchanged. We formalize five attack scenarios, including training on rewritten test examples and applying similarity-based decontamination. Across multiple-choice Q&A, text classification, and open-ended Q&A with Qwen and Gemma models, BenchDoor achieves positive benchmark gains for all 30 evaluated configurations, with held-out attack success rates of 82.33% to 100.00% at poisoning rates of 2% to 10%. Both objectives remain achievable after synthetic data generation and similarity-based decontamination, with attack success rates of at least 90.33%. These findings show how benchmarks can serve as attack vectors both through their own data and through training resources that deliver improvements on them.
Abstract:Machine learning benchmarks act as guideposts for model development and spur the creation of training datasets intended to improve benchmark scores. We introduce BenchDoor, a family of data poisoning attacks that exploits this incentive. Beyond preserving benign performance, BenchDoor requires benchmark gains that encourage adoption of data that also implants a backdoor. The adversary either manipulates a benchmark's training or test data, creating an adversarial benchmark, or supplies an independent training dataset while leaving the benchmark unchanged. We formalize five attack scenarios, including training on rewritten test examples and applying similarity-based decontamination. Across multiple-choice Q&A, text classification, and open-ended Q&A with Qwen and Gemma models, BenchDoor achieves positive benchmark gains for all 30 evaluated configurations, with held-out attack success rates of 82.33% to 100.00% at poisoning rates of 2% to 10%. Both objectives remain achievable after synthetic data generation and similarity-based decontamination, with attack success rates of at least 90.33%. These findings show how benchmarks can serve as attack vectors both through their own data and through training resources that deliver improvements on them.
Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI)
arXiv:2610.27533 [cs.CR]
(or arXiv:2610.27533v1 [cs.CR] for this version)
https://doi.org/10.48550/arXiv.2610.27533 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 272 words
Related coverage
First Unified Benchmark Red-Teams VQE Adversarial Robustness
Quantum Zeitgeist · Jul 24, 2026
ARB4WM: An Adversarial Robustness Benchmark for World Models in Continuous Control
arXiv · Jun 16, 2026
HARL-A: An Extensible Benchmark Framework for Heterogeneous Multi-Agent Adversarial Reinforcement Learning in IsaacLab
arXiv · Oct 7, 2026
AdversaRiskQA: An Adversarial Factuality Benchmark for High-Risk Domains
arXiv · Sep 30, 2026