aiqbench analysis
This study demonstrates that synthetically generated semantic descriptions can effectively supervise contrastive pretraining in small transformers, offering a scalable alternative to labor-intensive docstrings or costly execution traces for code representation tasks. The approach achieves performance parity with much larger zero-shot models and generalist LLMs on classification and retrieval benchmarks across C, C++, and Java. By discarding the synthetic text at inference, this method reduces computational overhead while maintaining high utility for code search and analysis tools.
Synthetic Semantic Supervision for Contrastive Code Representation Learning in Small Transformers: An Empirical Study
arXiv:2609.03702v1 Announce Type: new Abstract: General-purpose code embeddings power tools for code search, classification, and retrieval. Compact transformer encoders for code typically rely on either human-written docstrings (labor-intensive and inconsistent) or mined structural signals such as execution traces (setting-specific and costly to collect). We empirically study an alternative: contrastive pretraining of small encoders with synthetically generated natural-language descriptions emphasizing code functionality and intent, paired with code in a dual-encoder framework at training and discarded at inference. We benchmark this approach against pretraining-based baselines, generalist LLMs, and embedding-specific models on eight retrieval, classification, and generation tasks across C, C++, and Java. Synthetic semantic supervision yields statistically significant gains over pretraining baselines of the same inference-time size on five of eight tasks, with parity on two more; once fine-tuned, it matches or exceeds zero-shot models two orders of magnitude larger on classification, and it stays on par with execution-aware supervision at matched pretraining data, suggesting a scalable, effective alternative to existing code-representation paradigms.
Abstract:General-purpose code embeddings power tools for code search, classification, and retrieval. Compact transformer encoders for code typically rely on either human-written docstrings (labor-intensive and inconsistent) or mined structural signals such as execution traces (setting-specific and costly to collect). We empirically study an alternative: contrastive pretraining of small encoders with synthetically generated natural-language descriptions emphasizing code functionality and intent, paired with code in a dual-encoder framework at training and discarded at inference. We benchmark this approach against pretraining-based baselines, generalist LLMs, and embedding-specific models on eight retrieval, classification, and generation tasks across C, C++, and Java. Synthetic semantic supervision yields statistically significant gains over pretraining baselines of the same inference-time size on five of eight tasks, with parity on two more; once fine-tuned, it matches or exceeds zero-shot models two orders of magnitude larger on classification, and it stays on par with execution-aware supervision at matched pretraining data, suggesting a scalable, effective alternative to existing code-representation paradigms.
Artificial Intelligence (cs.AI)
arXiv:2609.03702 [cs.AI]
(or arXiv:2609.03702v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.03702 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 250 words
Related coverage
Invariant Pretraining for Robust Code Representations
arXiv · Aug 19, 2026
Thousand-dimensional structure
AI Alignment Forum · Jul 30, 2026
IConE: Batch Independent Collapse Prevention for Self-Supervised Representation Learning
arXiv · Jul 23, 2026
CAMMAR: Culture-Aware Matryoshka for Metaphorical Arabic Representations
arXiv · Jul 20, 2026