analiza aiqbench
This research introduces CDAE, a lightweight method designed to improve the robustness of pre-trained language models against semantic-preserving perturbations like synonym substitution and word dropout. By jointly optimizing contrastive and reconstruction objectives, the framework enhances embedding stability without sacrificing core semantic information, offering a practical solution for building more reliable NLP systems.
CDAE: Enhancing Perturbation Robustness in Pretrained Language Models with Contrastive Denoising
arXiv:2607.28236v1 Announce Type: cross Abstract: Pre-trained language models have significantly improved sentence representation learning, yet their embedding remain sensitive to semantic preserving textual perturbations such as synonym substitution, masking and word dropout. This work proposes a lightweight Contrastive Denoising Autoencoder (CDAE) that refines pre-trained BERT embedding by jointly optimizing contrastive and reconstruction objective to learn perturbation-invariant representation. We evaluate the proposed framework using multiple perturbation strategies with varying strengths and compare it against the original BERT embeddings and SimCSE. Experimental results show that CDAE consistently preserves higher embedding similarity under perturbations, with the improvements becoming more pronounced as framework effectively enhances representation stability while preserving semantic information, highlighting perturbation-invariant learning as a promising direction for improving sentence embeddings. The source code is publicly available at: https://github.com/ComputationIASBS/CDAE
Abstract:Pre-trained language models have significantly improved sentence representation learning, yet their embedding remain sensitive to semantic preserving textual perturbations such as synonym substitution, masking and word dropout. This work proposes a lightweight Contrastive Denoising Autoencoder (CDAE) that refines pre-trained BERT embedding by jointly optimizing contrastive and reconstruction objective to learn perturbation-invariant representation. We evaluate the proposed framework using multiple perturbation strategies with varying strengths and compare it against the original BERT embeddings and SimCSE. Experimental results show that CDAE consistently preserves higher embedding similarity under perturbations, with the improvements becoming more pronounced as framework effectively enhances representation stability while preserving semantic information, highlighting perturbation-invariant learning as a promising direction for improving sentence embeddings. The source code is publicly available at: this https URL
Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2607.28236 [cs.AI]
(or arXiv:2607.28236v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2607.28236 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 221 cuvinte
Subiecte conexe
Latent reasoning architectures would likely undermine CoT, our strongest oversight tool
AI Alignment Forum · 23 sept. 2026
A Robust and Explainable Transformer-Based Framework for Phishing Email Detection
arXiv · 3 iun. 2026
PIRAMID: Progress and Plans
LessWrong · 27 iul. 2026
Evaluation of Adversarial Robustness in Arabic Language Models
arXiv · 29 iul. 2026