analiza aiqbench
This research introduces S-GBT, a novel second-order training method that bounds Hessian elements to enhance certified robustness against word substitution attacks in NLP models. By incorporating curvature-aware regularization alongside standard gradient constraints, the approach achieves significant improvements in adversarial accuracy without sacrificing clean performance on benchmark datasets. The findings underscore the critical importance of controlling higher-order sensitivity for building more reliable language processing systems.
S-GBT: Smooth Growth Bound Tensor for Certified Robustness Against Word Substitution Attacks in NLP
arXiv:2606.13439v1 Announce Type: new Abstract: Despite recent progress in Natural Language Processing (NLP), models remain vulnerable to word substitution attacks. Most existing defenses focus on first order sensitivity and measure how much the output changes when the input is slightly perturbed. However, they ignore how this sensitivity evolves, which is described by curvature. When gradients vary sharply, models can still fail. This paper introduces the Smooth Growth Bound Tensor (S-GBT), a second order method that bounds the Hessian element-wise, for which we provide formal theoretical proofs on the resulting robustness bounds. A regularization term is added during training to minimize these bounds. This yields tighter certified robustness against word substitution attacks. The change in the output under word substitution is bounded by both a linear term and a quadratic term. S-GBT is derived for two architectures: Long Short-Term Memory (LSTM) and Convolutional Neural Networks (CNN). The method is integrated directly into the training objective. Its effectiveness is evaluated on multiple benchmark datasets. The results show that combining first and second order regularization improves certified robust accuracy by up to 23.4% compared to prior methods, while clean accuracy remains competitive. These findings indicate that controlling both the gradient and its variation is a promising direction for building more robust models.
Abstract:Despite recent progress in Natural Language Processing (NLP), models remain vulnerable to word substitution attacks. Most existing defenses focus on first order sensitivity and measure how much the output changes when the input is slightly perturbed. However, they ignore how this sensitivity evolves, which is described by curvature. When gradients vary sharply, models can still fail. This paper introduces the Smooth Growth Bound Tensor (S-GBT), a second order method that bounds the Hessian element-wise, for which we provide formal theoretical proofs on the resulting robustness bounds. A regularization term is added during training to minimize these bounds. This yields tighter certified robustness against word substitution attacks. The change in the output under word substitution is bounded by both a linear term and a quadratic term. S-GBT is derived for two architectures: Long Short-Term Memory (LSTM) and Convolutional Neural Networks (CNN). The method is integrated directly into the training objective. Its effectiveness is evaluated on multiple benchmark datasets. The results show that combining first and second order regularization improves certified robust accuracy by up to 23.4% compared to prior methods, while clean accuracy remains competitive. These findings indicate that controlling both the gradient and its variation is a promising direction for building more robust models.
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2606.13439 [cs.CL]
(or arXiv:2606.13439v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.13439 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 302 cuvinte
Subiecte conexe
Quantifying mixed-state entanglement via partial transpose and realignment moments
Quantum Journal · 19 aug. 2026
Automating Performance Bottleneck Identification with the TraceLens Agent
AMD · 30 sept. 2026
FoSeRL: Formal Sequential Robustness Certification for Reinforcement Learning Policies
arXiv · 6 oct. 2026
Affine-Profile Stabilizer Thresholds for Magic in Codeword-Stabilized Quantum Codes
arXiv · 27 aug. 2026