aiqbench analysis
This research bridges the gap between controllable classical graphics pipelines and generative AI by integrating physically based rendering principles directly into the latent space of diffusion models. By modifying the rendering equation to operate within variational autoencoder feature spaces, the method enables precise control over light transport, geometry, and lighting for content generation. This advancement offers a significant step toward merging high-fidelity physics-based simulation with the flexibility of modern generative AI.
Physically Based Rendering in the Latent Space
arXiv:2609.21054v1 Announce Type: cross Abstract: Image diffusion models have shown impressive image generation capabilities but are often hard to control, in contrast to classical computer graphics pipelines such as physically based rendering. However, we observe that there is a bridge between light transport phenomena and the distribution of latent space values produced by such models. Thus, we introduce physically based rendering in the feature space learned by the variational autoencoders in generative models, enabling light transport simulation in the latent space. This allows us to leverage physically based rendering techniques to output latent maps for physically guided content generation. We propose modifications to the rendering equation, which, when paired with a differentiable renderer, can yield an optimal set of scene parameters that require only minimal refinement to accurately render into the pretrained latent space. We train our method on a single rendered image, and then demonstrate the generalization of the method to scene geometry changes, lighting changes, and camera view changes.
Abstract:Image diffusion models have shown impressive image generation capabilities but are often hard to control, in contrast to classical computer graphics pipelines such as physically based rendering. However, we observe that there is a bridge between light transport phenomena and the distribution of latent space values produced by such models. Thus, we introduce physically based rendering in the feature space learned by the variational autoencoders in generative models, enabling light transport simulation in the latent space. This allows us to leverage physically based rendering techniques to output latent maps for physically guided content generation. We propose modifications to the rendering equation, which, when paired with a differentiable renderer, can yield an optimal set of scene parameters that require only minimal refinement to accurately render into the pretrained latent space. We train our method on a single rendered image, and then demonstrate the generalization of the method to scene geometry changes, lighting changes, and camera view changes.
15 pages, 12 figures. Pacific Graphics 2026, Journal Track (Computer Graphics Forum). Code: this https URL
Graphics (cs.GR); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2609.21054 [cs.GR]
(or arXiv:2609.21054v1 [cs.GR] for this version)
https://doi.org/10.48550/arXiv.2609.21054 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
https://doi.org/10.1111/cgf.70633 Focus to learn more DOI(s) linking to related resources
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 281 words
Related coverage
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · Sep 25, 2026
[AINews] The Future of Latent Space
Latent Space · Sep 25, 2026
World Models and Yann LeCun’s Bet Against Large Language Models
Quantum Zeitgeist · Jul 17, 2026
Developing Healthcare Robotics with GPU-Native Medical Physics Simulation
NVIDIA · Jul 28, 2026