aiqbench analysis
This research introduces OASIS, a novel framework that optimizes the sequence of attacks in hard-label black-box scenarios against large language models. By treating attacker composition as an optimization target rather than a manual choice, the method achieves superior success rates while balancing perturbation levels across diverse datasets and victim models. This work advances the field of AI safety by providing a more robust approach to evaluating and defending LLMs against sophisticated adversarial sequences.
OASIS: Optimizing Attacker Sequences for Hard-Label Black-Box Text Attacks
arXiv:2608.29568v1 Announce Type: cross Abstract: Different attack methods follow different search trajectories, they succeed on different subsets of samples, whereas existing hard-label black-box text attacks mainly focus on improving individual attackers or manually combining them. We present {\OURS}, a method for optimizing attacker sequences in hard-label black-box text attacks. {\OURS} first performs a one-time bi-objective attack chain search over candidate sequences to balance attack success rate and perturbation, and then reuses the selected fixed global chain during attack chain execution. Experiments across multiple datasets, victim models, and large language models show that {\OURS} consistently outperforms strong standalone baselines and simple manually constructed chains. These results suggest that attacker composition is not merely an implementation choice, but a practical optimization target for improving hard-label black-box text attacks.
Abstract:Different attack methods follow different search trajectories, they succeed on different subsets of samples, whereas existing hard-label black-box text attacks mainly focus on improving individual attackers or manually combining them. We present {\OURS}, a method for optimizing attacker sequences in hard-label black-box text attacks. {\OURS} first performs a one-time bi-objective attack chain search over candidate sequences to balance attack success rate and perturbation, and then reuses the selected fixed global chain during attack chain execution. Experiments across multiple datasets, victim models, and large language models show that {\OURS} consistently outperforms strong standalone baselines and simple manually constructed chains. These results suggest that attacker composition is not merely an implementation choice, but a practical optimization target for improving hard-label black-box text attacks.
Qian Chen and Shiliang Xiao contributed equally. Yuzhi Liang is the corresponding author
Cryptography and Security (cs.CR); Computation and Language (cs.CL)
arXiv:2608.29568 [cs.CR]
(or arXiv:2608.29568v1 [cs.CR] for this version)
https://doi.org/10.48550/arXiv.2608.29568 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 233 words
