State Key Lab of CAD & CG, Zhejiang University
A fully automated object reconstruction pipeline is crucial for digital content creation. While the area of 3D reconstruction has witnessed profound developments, the removal of background to obtain a clean object model still relies on different forms of manual labor, such as bounding box labeling, mask annotations, and mesh manipulations. In this paper, we propose a novel framework named AutoRecon for the automated discovery and reconstruction of an object from multi-view images. We demonstrate that foreground objects can be robustly located and segmented from SfM point clouds by leveraging self-supervised 2D vision transformer features. Then, we reconstruct decomposed neural scene representations with dense supervision provided by the decomposed point clouds, resulting in accurate object reconstruction and segmentation. Experiments on the DTU, BlendedMVS and CO3D-V2 datasets demonstrate the effectiveness and robustness of AutoRecon.
Given an object-centric video, we achieve coarse decomposition by segmenting the salient foreground object from a semi-dense SfM point cloud, with pointwise-aggregated 2D DINO features. Then we train a decomposed neural scene representation from multi-view images with the help of coarse decomposition results to reconstruct foreground objects and render multi-view consistent high-quality foreground masks.
* please drag & scroll the "Coarse Decomposition" figure to see surrounding SfM point clouds.
* we show reconstructed mesh with baked ambient occlusion.
@inproceedings{wang2023autorecon,
title={AutoRecon: Automated 3D Object Discovery and Reconstruction},
author={Wang, Yuang and He, Xingyi and Peng, Sida and Lin, Haotong and Bao, Hujun and Zhou, Xiaowei},
booktitle={CVPR},
year={2023}
}