Animatable Neural Radiance Fields for Human Body Modeling


Sida Peng1*, Junting Dong1*, Qianqian Wang2, Shangzhan Zhang1, Qing Shuai1, Hujun Bao1, Xiaowei Zhou1

1State Key Lab of CAD & CG, Zhejiang University    2Cornell University
* denotes equal contribution

Abstract


Animatable NeRF can reconstruct an animatable human model from a sparse multi-view video.

This paper addresses the challenge of reconstructing an animatable human model from a multi-view video. Some recent works have proposed to decompose a non-rigidly deforming scene into a canonical neural radiance field and a set of deformation fields that map observation-space points to the canonical space, thereby enabling them to learn the dynamic scene from images. However, they represent the deformation field as translational vector field or SE(3) field, which makes the optimization highly under-constrained. Moreover, these representations cannot be explicitly controlled by input motions. Instead, we introduce neural blend weight fields to produce the deformation fields. Based on the skeleton-driven deformation, blend weight fields are used with 3D human skeletons to generate observation-to-canonical and canonical-to-observation correspondences. Since 3D human skeletons are more observable, they can regularize the learning of deformation fields. Moreover, the learned blend weight fields can be combined with input skeletal motions to generate new deformation fields to animate the human model. Experiments show that our approach significantly outperforms recent human synthesis methods. The code will be available at https://zju3dv.github.io/animatable_nerf/.


Overview video



Comparison on novel pose synthesis



Comparison on novel view synthesis



Reposing the reconstructed geometry



Citation


@article{peng2021animatable,
  title={Animatable Neural Radiance Fields for Human Body Modeling},
  author={Peng, Sida and Dong, Junting and Wang, Qianqian and Zhang, Shangzhan and Shuai, Qing and Bao, Hujun and Zhou, Xiaowei},
  journal={arXiv preprint arXiv:2105.02872},
  year={2021}
}