aiqbench analysis
This research introduces VOLTA, a novel framework that enhances generative diversity in Transformer models by integrating Variational Autoencoders through cross-attention mechanisms rather than standard concatenation. By incorporating input-independent latent codes inspired by InfoGAN, the method successfully expands the exploration of the latent space without compromising generation quality across multiple natural language tasks. The work addresses a critical limitation in current state-of-the-art models where high fidelity often comes at the expense of output variety.
VOLTA: Improving Generative Diversity by Variational Mutual Information Maximizing Autoencoder
arXiv:2307.00852v3 Announce Type: replace Abstract: The natural language generation domain has witnessed great success thanks to Transformer models. Although they have achieved state-of-the-art generative quality, they often neglect generative diversity. Prior attempts to tackle this issue suffer from either low model capacity or over-complicated architectures. Some recent methods employ the VAE framework to enhance diversity, but their latent variables fully depend on the input context, restricting exploration of the latent space. In this paper, we introduce VOLTA, a framework that elevates generative diversity by bridging Transformer with VAE via a more effective cross-attention-based connection, departing from conventional embedding concatenation or summation. Additionally, we propose integrating InfoGAN-style latent codes to enable input-independent variability, further diversifying the generation. Moreover, our framework accommodates discrete inputs alongside its existing support for continuous inputs. We perform comprehensive experiments with two types of Transformers on six datasets from three different NLG tasks to show that our approach can significantly improve generative diversity while maintaining generative quality.
Abstract:The natural language generation domain has witnessed great success thanks to Transformer models. Although they have achieved state-of-the-art generative quality, they often neglect generative diversity. Prior attempts to tackle this issue suffer from either low model capacity or over-complicated architectures. Some recent methods employ the VAE framework to enhance diversity, but their latent variables fully depend on the input context, restricting exploration of the latent space. In this paper, we introduce VOLTA, a framework that elevates generative diversity by bridging Transformer with VAE via a more effective cross-attention-based connection, departing from conventional embedding concatenation or summation. Additionally, we propose integrating InfoGAN-style latent codes to enable input-independent variability, further diversifying the generation. Moreover, our framework accommodates discrete inputs alongside its existing support for continuous inputs. We perform comprehensive experiments with two types of Transformers on six datasets from three different NLG tasks to show that our approach can significantly improve generative diversity while maintaining generative quality.
15 pages. Published in Findings of the Association for Computational Linguistics: NAACL 2024
Computation and Language (cs.CL)
arXiv:2307.00852 [cs.CL]
(or arXiv:2307.00852v3 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2307.00852 Focus to learn more arXiv-issued DOI via DataCite
https://doi.org/10.18653/v1/2024.findings-naacl.26 Focus to learn more DOI(s) linking to related resources
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 271 words
Related coverage
FlowEdit: Information-Theoretic Control of LLM Reasoning Flows for Ill-posed Problems Involving Conflicts
arXiv · Jul 24, 2026
What's Continual Learning, and Why Might We Expect To See It In Advanced LLM Agents?
LessWrong · Jun 12, 2026
2026 BAIR Graduate Showcase
BAIR Blog · Jul 1, 2026
Kankeu and Colleagues Present Hybrid Variational Autoencoder for Neural Topic Modeling
Quantum Zeitgeist · Jun 17, 2026