POET: Preference Optimization for Enhanced Text-to-Image Generation

arXiv:2510.12041v3 Announce Type: replace Abstract: Recent advances in text-to-image (T2I) generation have achieved impressive results, yet existing models often struggle with simple or underspecified user prompts due to a distributional gap with their descriptive training captions. This frequently leads to suboptimal image-text alignment, aesthetics, and overall visual quality. To bridge this gap, we propose POET (\textbf{P}reference \textbf{O}ptimization for \textbf{E}nhanced \textbf{T}ext-to-Image generation), an automated prompt rewriting framework that leverages large language models (LLMs) to refine user inputs before feeding them into frozen T2I backbones. POET introduces a carefully designed composite reward system and an iterative Direct Preference Optimization (DPO) training pipeline, enabling the rewriter to learn model-preferred prompt structures directly from multimodal feedback without requiring costly high-quality supervised fine-tuning (SFT) data. Extensive evaluations across diverse T2I models and benchmarks show that our prompt rewriter consistently improves image-text alignment, visual quality, and aesthetics, outperforming strong baselines. Furthermore, we demonstrate strong transferability by showing that a rewriter trained on one T2I backbone generalizes effectively to others without needing to be retrained. These findings highlight that POET is an effective, robust, and practical model-agnostic strategy for improving T2I systems.

Abstract:Recent advances in text-to-image (T2I) generation have achieved impressive results, yet existing models often struggle with simple or underspecified user prompts due to a distributional gap with their descriptive training captions. This frequently leads to suboptimal image-text alignment, aesthetics, and overall visual quality. To bridge this gap, we propose POET (\textbf{P}reference \textbf{O}ptimization for \textbf{E}nhanced \textbf{T}ext-to-Image generation), an automated prompt rewriting framework that leverages large language models (LLMs) to refine user inputs before feeding them into frozen T2I backbones. POET introduces a carefully designed composite reward system and an iterative Direct Preference Optimization (DPO) training pipeline, enabling the rewriter to learn model-preferred prompt structures directly from multimodal feedback without requiring costly high-quality supervised fine-tuning (SFT) data. Extensive evaluations across diverse T2I models and benchmarks show that our prompt rewriter consistently improves image-text alignment, visual quality, and aesthetics, outperforming strong baselines. Furthermore, we demonstrate strong transferability by showing that a rewriter trained on one T2I backbone generalizes effectively to others without needing to be retrained. These findings highlight that POET is an effective, robust, and practical model-agnostic strategy for improving T2I systems.

Computation and Language (cs.CL)

arXiv:2510.12041 [cs.CL]

(or arXiv:2510.12041v3 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2510.12041 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

analiza aiqbench

POET: Preference Optimization for Enhanced Text-to-Image Generation

Subiecte conexe