Painting 3D Nature in 2D: View Synthesis of Natural Scenes from a Single Semantic Mask

CVPR 2023

Shangzhan Zhang1, Sida Peng1, Tianrun Chen1, Linzhan Mou1, Haotong Lin1,
Kaicheng Yu2, Yiyi Liao1, Xiaowei Zhou 1
1Zhejiang University     2Alibaba Group
Paper Supplementary Material Code

Abstract

We introduce a novel approach that takes a single semantic mask as input to synthesize multi-view consistent color images of natural scenes, trained with a collection of single images from the Internet. Prior works on 3D-aware image synthesis either require multi-view supervision or learning category-level prior for specific classes of objects, which can hardly work for natural scenes. Our key idea to solve this challenging problem is to use a semantic field as the intermediate representation, which is easier to reconstruct from an input semantic mask and then translate to a radiance field with the assistance of off-the-shelf semantic image synthesis models. Experiments show that our method outperforms baseline methods and produces photorealistic, multi-view consistent videos of a variety of natural scenes.

Problem Setting

Gallery

BibTeX

@inproceedings{Zhang2023Painting3N,
  title={Painting 3D Nature in 2D: View Synthesis of Natural Scenes from a Single Semantic Mask},
  author={Shangzhan Zhang and Sida Peng and Tianrun Chen and Linzhan Mou and Haotong Lin and Kaicheng Yu and Yiyi Liao and Xiaowei Zhou},
  booktitle={CVPR},
  year={2023}
}