analiza aiqbench
LLM2Vec-Gen introduces a novel self-supervised method that generates embeddings directly within an LLM's output space, preserving critical capabilities like safety alignment and reasoning without fine-tuning the model backbone. By optimizing special tokens to compress potential responses into fixed-length vectors using only unlabeled data, this approach achieves state-of-the-art performance on embedding benchmarks while offering interpretable results decodable back into text. This advancement represents a significant efficiency gain for retrieval-augmented generation systems that require high-quality semantic representations with minimal computational overhead.
LLM2Vec-Gen: Generative Embeddings from Large Language Models
arXiv:2603.10913v3 Announce Type: replace Abstract: Fine-tuning LLM-based text embedders via contrastive learning maps inputs and outputs into a new representational space, discarding the LLM's output semantics. We propose LLM2Vec-Gen, a self-supervised alternative that instead produces embeddings directly in the LLM's output space by learning to represent the model's potential response. Specifically, trainable special tokens are appended to the input and optimized to compress the LLM's own response into a fixed-length embedding, guided by an unsupervised embedding teacher and a reconstruction objective. Crucially, the LLM backbone remains frozen and training requires only unlabeled queries. LLM2Vec-Gen achieves state-of-the-art self-supervised performance on the Massive Text Embedding Benchmark (MTEB), improving by 8.8% over the unsupervised embedding teacher. Since the embeddings preserve the LLM's response-space semantics, they inherit capabilities such as safety alignment (up to 22.6% reduction in harmful content retrieval) and reasoning (up to 35.6% improvement on reasoning-intensive retrieval). Finally, the learned embeddings are also interpretable: they can be decoded back into text to reveal their semantic content. Our code, data, and pretrained models are publicly available at https://github.com/McGill-NLP/llm2vec-gen.
Abstract:Fine-tuning LLM-based text embedders via contrastive learning maps inputs and outputs into a new representational space, discarding the LLM's output semantics. We propose LLM2Vec-Gen, a self-supervised alternative that instead produces embeddings directly in the LLM's output space by learning to represent the model's potential response. Specifically, trainable special tokens are appended to the input and optimized to compress the LLM's own response into a fixed-length embedding, guided by an unsupervised embedding teacher and a reconstruction objective. Crucially, the LLM backbone remains frozen and training requires only unlabeled queries. LLM2Vec-Gen achieves state-of-the-art self-supervised performance on the Massive Text Embedding Benchmark (MTEB), improving by 8.8% over the unsupervised embedding teacher. Since the embeddings preserve the LLM's response-space semantics, they inherit capabilities such as safety alignment (up to 22.6% reduction in harmful content retrieval) and reasoning (up to 35.6% improvement on reasoning-intensive retrieval). Finally, the learned embeddings are also interpretable: they can be decoded back into text to reveal their semantic content. Our code, data, and pretrained models are publicly available at this https URL.
Computation and Language (cs.CL)
arXiv:2603.10913 [cs.CL]
(or arXiv:2603.10913v3 [cs.CL] for this version)
Text integral sub licența deschisă a publicației · 184 cuvinte
Subiecte conexe
Article: Your Next DSL Author is a Language Model
InfoQ · 16 sept. 2026
Presentation: From Models to Agents: Building Context-Aware Consumer AI at Scale at DoorDash
InfoQ · 15 aug. 2026
From code to diagrams: Agentic architecture documentation with Amazon Bedrock AgentCore
AWS · 2 sept. 2026
Selecting a vector store for Amazon Bedrock Knowledge Bases
AWS · 17 sept. 2026