Animatable Neural Implicit Surfaces for Creating Avatars from Videos


Sida Peng1, Shangzhan Zhang1, Zhen Xu1, Chen Geng1, Boyi Jiang2, Hujun Bao1, Xiaowei Zhou1

1State Key Lab of CAD & CG, Zhejiang University    2Image Derivative Inc

Abstract


This paper is an extension of Animatable NeRF, which has better reconstruction and rendering results.

This paper aims to reconstruct an animatable human model from a video of very sparse camera views. Some recent works represent human geometry and appearance with neural radiance fields and utilize parametric human models to produce deformation fields for animation, which enables them to recover detailed 3D human models from videos. However, their reconstruction results tend to be noisy due to the lack of surface constraints on radiance fields. Moreover, as they generate the human appearance in 3D space, their rendering quality heavily depends on the accuracy of deformation fields. To solve these problems, we propose Animatable Neural Implicit Surface (AniSDF), which models the human geometry with a signed distance field and defers the appearance generation to the 2D image space with a 2D neural renderer. The signed distance field naturally regularizes the learned geometry, enabling the high-quality reconstruction of human bodies, which can be further used to improve the rendering speed. Moreover, the 2D neural renderer can be learned to compensate for geometric errors, making the rendering more robust to inaccurate deformations. Experiments on several datasets show that the proposed approach outperforms recent human reconstruction and synthesis methods by a large margin.


Reconstruction results


Comparison on 3D reconstruction from monocular videos
More reconstruction results

Rendering results


Comparison on novel view synthesis
Comparison on novel pose synthesis
Novel pose synthesis under complex human poses

Ablation studies


Non-rigid deformations captured by the displacement fields
Ablation study on canonical-space viewing direction
Ablation study on neural feature fields

Citation


@article{peng2022animatable,
  title={Animatable Neural Implict Surfaces for Creating Avatars from Videos},
  author={Peng, Sida and Zhang, Shangzhan and Xu, Zhen and Geng, Chen and Jiang, Boyi and Bao, Hujun and Zhou, Xiaowei},
  journal={arXiv preprint arXiv:2203.08133},
  year={2022}
}