analiza aiqbench
This research investigates how fine-tuning visual encoders for reconstruction alters the geometry of high-dimensional latent spaces, ultimately reducing their effective dimensionality and hindering standard flow matching optimization. The authors demonstrate that switching to clean data parameterization ($\boldsymbol{x}_{0}$-prediction) allows models to focus on the underlying signal manifold, significantly improving text-to-image generation performance without relying on inefficient noise fitting.
On the Diffusibility of High-Dimensional Latents
arXiv:2609.28473v1 Announce Type: cross Abstract: Representation Autoencoders (RAEs) enable diffusion models to operate in the feature spaces of pretrained visual encoders. However, many off-the-shelf encoders are not optimized for faithful reconstruction, discarding fine-grained visual details. As expected, finetuning these encoders for image reconstruction recovers such details. However, perhaps counterintuitively, this procedure reduces the effective dimensionality of the resulting representation, and the altered geometry has downstream effects on generation. Specifically, we show that using the standard velocity prediction in flow matching in this high-dimensional space requires the model to fit orthogonal noise directions outside the low-dimensional signal manifold, making optimization inefficient. This motivates using the clean data parameterization ($\boldsymbol{x}_{0}$-prediction) instead, which focuses learning on the underlying signal manifold. Across experiments with multiple strong-reconstruction encoders, we show that $\boldsymbol{x}_{0}$-prediction consistently improves text-to-image generation performance.
Abstract:Representation Autoencoders (RAEs) enable diffusion models to operate in the feature spaces of pretrained visual encoders. However, many off-the-shelf encoders are not optimized for faithful reconstruction, discarding fine-grained visual details. As expected, finetuning these encoders for image reconstruction recovers such details. However, perhaps counterintuitively, this procedure reduces the effective dimensionality of the resulting representation, and the altered geometry has downstream effects on generation. Specifically, we show that using the standard velocity prediction in flow matching in this high-dimensional space requires the model to fit orthogonal noise directions outside the low-dimensional signal manifold, making optimization inefficient. This motivates using the clean data parameterization ($\boldsymbol{x}_{0}$-prediction) instead, which focuses learning on the underlying signal manifold. Across experiments with multiple strong-reconstruction encoders, we show that $\boldsymbol{x}_{0}$-prediction consistently improves text-to-image generation performance.
Accepted to ECCV 2026. Project page: this https URL
Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
arXiv:2609.28473 [cs.CV]
(or arXiv:2609.28473v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2609.28473 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 236 cuvinte
Subiecte conexe
Intrinsic-Hybrid Latent Diffusion Models for Generative Modeling on Unknown Manifolds
arXiv · 6 aug. 2026
Data assimilation for subsurface flow using latent diffusion model parameterization: performance of ensemble-Kalman and Monte Carlo techniques
arXiv · 10 iun. 2026
One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation
Apple · 15 iul. 2026
BSTabDiff: Block-Subunit Diffusion Priors for High-Dimensional Tabular Data Generation
arXiv · 9 iun. 2026