aiqbench analysis
This research addresses data sparsity in traditional neural topic models by distilling soft labels from language models to create contextually enriched reconstruction signals. The approach yields higher-quality topics with improved coherence and assignment accuracy, while also demonstrating superior performance in identifying semantically similar documents for retrieval tasks. By leveraging LM hidden states and next-token probabilities, the method offers a practical upgrade for applications requiring robust thematic analysis and document clustering.
Improving Topic Modeling by Distilling Soft Labels from Language Models
arXiv:2602.17907v3 Announce Type: replace Abstract: Traditional neural topic models are typically optimized by reconstructing the document's Bag-of-Words (BoW) representations, overlooking contextual information and struggling with data sparsity. In this work, we introduce a novel topic model training framework by Distilling Soft Labels (DSL) from Language Models (LMs). To construct the contextually enriched reconstruction signals, we project the next token probabilities, conditioned on a specialized prompt, onto a pre-defined vocabulary, and train the topic models to reconstruct the soft labels using the LM hidden states. This produces higher-quality topics that are more closely aligned with the underlying thematic structure of the corpus. Extensive experiments demonstrate that DSL achieves substantial improvements in topic coherence and assignment accuracy over existing baselines. Additionally, we also introduce a retrieval-based metric, which shows that our approach significantly outperforms existing methods in identifying semantically similar documents, highlighting its effectiveness for retrieval-oriented applications.
Abstract:Traditional neural topic models are typically optimized by reconstructing the document's Bag-of-Words (BoW) representations, overlooking contextual information and struggling with data sparsity. In this work, we introduce a novel topic model training framework by Distilling Soft Labels (DSL) from Language Models (LMs). To construct the contextually enriched reconstruction signals, we project the next token probabilities, conditioned on a specialized prompt, onto a pre-defined vocabulary, and train the topic models to reconstruct the soft labels using the LM hidden states. This produces higher-quality topics that are more closely aligned with the underlying thematic structure of the corpus. Extensive experiments demonstrate that DSL achieves substantial improvements in topic coherence and assignment accuracy over existing baselines. Additionally, we also introduce a retrieval-based metric, which shows that our approach significantly outperforms existing methods in identifying semantically similar documents, highlighting its effectiveness for retrieval-oriented applications.
22 pages, 5 figures. Camera-ready version for ICML 2026
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2602.17907 [cs.CL]
(or arXiv:2602.17907v3 [cs.CL] for this version)
Full text under the publisher's open licence · 164 words
Related coverage
DSL-Topic: Improving Topic Modeling by Distilling Soft Labelsfrom Language Models
arXiv · Jun 4, 2026
[AINews] not much happened today
Latent Space · Sep 10, 2026
Scaling and Distilling Text Embeddings for Better Diffusibility
arXiv · Oct 2, 2026
Does distilling Claude carry the persona with it?
LessWrong · Jul 24, 2026