IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering
Under Unknown Illumination

Xi Chen1, Sida Peng1, Dongchen Yang1, Yuan Liu2, Bowen Pan3, Chengfei Lv3, Xiaowei Zhou1

1 Zhejiang University   2 The University of Hong Kong   3 Tao Technology Department, Alibaba Group  


Our approach can recover object materials from any images and enable single-view image relighting.

This paper aims to recover object materials from posed images captured under an unknown static lighting condition. Recent methods solve this task by representing materials using neural networks and optimizing model parameters through differentiable physically based rendering. However, due to the coupling between object geometry, materials, and environment lighting, there is inherent ambiguity during the inverse rendering process, preventing previous methods from obtaining accurate results. To overcome this ill-posed problem, our key idea is to learn the material prior with a generative model for regularizing the optimization process. We observe that the general rendering equation can be split into diffuse and specular shading terms, and thus formulate the material prior as diffusion models of albedo and specular. Thanks to this design, our model can be trained using the existing abundant 3D object data, and naturally acts as a versatile tool to resolve the ambiguity when recovering many material representations from RGB images. In addition, we develop a coarse-to-fine training strategy that leverages estimated materials to guide diffusion models to produce multi-view consistent constraints, leading to more stable and accurate results. Extensive experiments on real-world and synthetic datasets demonstrate that our approach achieves state-of-the-art performance on material recovery.

Comparasion with SOTA Methods

More Delighting Results

3D Inverse Rendering Pipeline

Comparison with baseline methods (Comming Soon...)

Real-world results (Comming Soon...)


  title     = {IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination},
  author    = {Xi, Chen and Sida, Peng and Dongchen, Yang and Yuan, Liu and Bowen, Pan and Chengfei, Lv and Xiaowei, Zhou.},
  journal   = {arxiv: 2404.11593},
  year      = {2024},