Neural Body: Implicit Neural Representations with Structured Latent Codes
for Novel View Synthesis of Dynamic Humans

TPAMI 2023, CVPR 2021 (Best Paper Candidate)

Sida Peng1, Chen Geng1, Yuanqing Zhang1, Yinghao Xu2, Qianqian Wang3, Qing Shuai1, Xiaowei Zhou1, Hujun Bao1

1State Key Lab of CAD & CG, Zhejiang University    2The Chinese University of Hong Kong    3Cornell University

Journal Paper

Conference Paper

Code

Data

Supplementary

Abstract

Neural Body can reconstruct a moving human from a monocular video.

This paper addresses the challenge of novel view synthesis for a human performer from a very sparse set of camera views. Some recent works have shown that learning implicit neural representations of 3D scenes achieves remarkable view synthesis quality given dense input views. However, the representation learning will be ill-posed if the views are highly sparse. To solve this ill-posed problem, our key idea is to integrate observations over video frames. To this end, we propose Neural Body, a new human body representation which assumes that the learned neural representations at different frames share the same set of latent codes anchored to a deformable mesh, so that the observations across frames can be naturally integrated. The deformable mesh also provides geometric guidance for the network to learn 3D representations more efficiently. To evaluate our approach, we create a multi-view dataset named ZJU-MoCap that captures performers with complex motions. Experiments on ZJU-MoCap show that our approach outperforms prior works by a large margin in terms of novel view synthesis quality. We also demonstrate the capability of our approach to reconstruct a moving person from a monocular video on the People-Snapshot dataset.


Overview video


Bullet time effects on street dance


Results on Human3.6M dataset


Comparison with state-of-the-art methods on sparse multi-view videos

Novel view synthesis of dynamic human

Novel view synthesis of frame 1

3D reconstruction


Results on monocular videos


Citation


  @article{peng2023implicit,
    title={Implicit Neural Representations with Structured Latent Codes for Human Body Modeling},
    author={Peng, Sida and Geng, Chen and Zhang, Yuanqing and Xu, Yinghao and Wang, Qianqian and Shuai, Qing and Zhou, Xiaowei and Bao, Hujun},
    journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
    year={2023},
    publisher={IEEE}
  }
  @inproceedings{peng2021neural,
    title={Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans},
    author={Peng, Sida and Zhang, Yuanqing and Xu, Yinghao and Wang, Qianqian and Shuai, Qing and Bao, Hujun and Zhou, Xiaowei},
    booktitle={CVPR},
    year={2021}
  }