Recently, Neural Radiance Fields (NeRF) has exhibited significant success in novel view synthesis, surface reconstruction, etc. However, since no physical reflection is considered in its rendering pipeline, NeRF mistakes the reflection in the mirror as a separate virtual scene, leading to the inaccurate reconstruction of the mirror and multi-view inconsistent reflections in the mirror. In this paper, we present a novel neural rendering framework, named MirrorNeRF, which is able to learn accurate geometry and reflection of the mirror and support various scene manipulation applications with mirrors, such as adding new objects or mirrors into the scene and synthesizing the reflections of these new objects in mirrors, controlling mirror roughness, etc. To achieve this goal, we propose a unified radiance field by introducing the reflection probability and tracing rays following the light transport model of Whitted Ray Tracing, and also develop several techniques to facilitate the learning process. Experiments and comparisons on both synthetic and real datasets demonstrate the superiority of our method.
Framework Overview
We trace the rays physically in the scene and learn a unified radiance field of the scene with the mirror. The neural field takes as input spatial location $\mathbf{x}$, view direction $\mathbf{d}$, and outputs the volume density $\hat{\sigma}$, radiance $\hat{\mathbf{c}}$, surface normal $\hat{\mathbf{n}}$ and reflection probability $\hat{m}$. The final color is blended by the color of the camera ray and the reflected ray based on the reflection probability.
Applications
Mirror-NeRF supports integrating new mirrors into the original scene by tracing the reflected rays at the mirror recursively. Mirror-NeRF enables the synthesis of novel views involving inter-reflection between the newly positioned mirror and the original mirror.
Mirror-NeRF supports the composition of multiple neural radiance fields and synthesizes new reflections of the compositional scenes in the mirror. This application might be of great use in VR and AR.
Mirror-NeRF supports reflection substitution by tracing the reflected rays in the new scene.
Mirror-NeRF supports controlling the roughness of the mirror by simulating the microfacets theory, since the geometry of the mirror is learned and ray tracing is supported.
@inproceedings{zeng2023mirror-nerf,
title={Mirror-NeRF: Learning Neural Radiance Fields for Mirrors with Whitted-Style Ray Tracing},
author={Zeng, Junyi and Bao, Chong and Chen, Rui and Dong, Zilong and Zhang, Guofeng and Bao, Hujun and Cui, Zhaopeng},
booktitle={Proceedings of the 31st ACM International Conference on Multimedia},
pages={4606--4615},
year={2023}
}