DiffPhysCam: Differentiable Physics-Based Camera Simulation for Inverse Rendering and Embodied AI

arXiv:2508.08831v2 Announce Type: replace-cross Abstract: Generating synthetic images that closely mimic those from real cameras is instrumental in training visual models and enabling end-to-end visuomotor learning. We introduce DiffPhysCam, a differentiable camera simulator designed to support robotics and embodied AI applications by enabling gradient-based optimization in visual perception pipelines. Differentiable rendering also allows inverse reconstruction of real-world scenes as digital twins, facilitating simulation-based robotics training. Existing virtual cameras offer limited control over intrinsic settings, have difficulty capturing optical artifacts, and lack tunable calibration parameters. DiffPhysCam addresses these limitations through a multi-stage pipeline that provides fine-grained control over camera settings, models key optical effects such as defocus blur, and supports calibration with real-world data. It enables both forward rendering for image synthesis and inverse rendering for 3D scene reconstruction, including mesh and material texture optimization. We show that DiffPhysCam enhances robotic perception performance in synthetic image tasks. As an illustrative example, we create a digital twin of a real-world scene using inverse rendering and use it to set up a virtual experiment in a multi-physics simulation, in which we demonstrate navigation of an autonomous ground vehicle using images generated by DiffPhysCam. The code, data, and output results associated with this paper are available online for reproducibility at Chen, Bo-Hsun (2026a,b,c).

Abstract:Generating synthetic images that closely mimic those from real cameras is instrumental in training visual models and enabling end-to-end visuomotor learning. We introduce DiffPhysCam, a differentiable camera simulator designed to support robotics and embodied AI applications by enabling gradient-based optimization in visual perception pipelines. Differentiable rendering also allows inverse reconstruction of real-world scenes as digital twins, facilitating simulation-based robotics training. Existing virtual cameras offer limited control over intrinsic settings, have difficulty capturing optical artifacts, and lack tunable calibration parameters. DiffPhysCam addresses these limitations through a multi-stage pipeline that provides fine-grained control over camera settings, models key optical effects such as defocus blur, and supports calibration with real-world data. It enables both forward rendering for image synthesis and inverse rendering for 3D scene reconstruction, including mesh and material texture optimization. We show that DiffPhysCam enhances robotic perception performance in synthetic image tasks. As an illustrative example, we create a digital twin of a real-world scene using inverse rendering and use it to set up a virtual experiment in a multi-physics simulation, in which we demonstrate navigation of an autonomous ground vehicle using images generated by DiffPhysCam. The code, data, and output results associated with this paper are available online for reproducibility at Chen, Bo-Hsun (2026a,b,c).

Graphics (cs.GR); Computer Vision and Pattern Recognition (cs.CV); Robotics (cs.RO)

arXiv:2508.08831 [cs.GR]

(or arXiv:2508.08831v2 [cs.GR] for this version)

https://doi.org/10.48550/arXiv.2508.08831 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

analiza aiqbench

DiffPhysCam: Differentiable Physics-Based Camera Simulation for Inverse Rendering and Embodied AI

Subiecte conexe