RealMe : Immersive Volumetric Video from Minimal Captures

Chenxu Zhou¹ Zhiyuan Yu^1,† Yudong Jin¹ Yinji Shentu¹ Yu Zhang² Jiaming Sun² Xiaowei Zhou¹

¹ State Key Laboratory of CAD&CG, Zhejiang University ² 4DV.ai

† Corresponding author

SIGGRAPH Asia 2026

Paper

arXiv

Code

Data

TL;DR

RealMe turns sparse, low-resolution multi-view captures into 2K, 4D-consistent volumetric video.

We adapt a high-resolution diffusion prior for 4D enhancement, with frequency guidance to recover fine details.

We create RealVV, a self-captured multi-view dataset for challenging capture settings.

Demo Video

Video loading

Method

Pipeline overview. From sparse, low-resolution (720p) multi-view captures, RealMe builds a coarse 4D representation and creates auxiliary videos from novel-view renders and crops of the inputs. Conditioned on these auxiliary videos, a high-resolution video diffusion model generates enhanced videos that are used together with the original captures to optimize the final 4D representation.

RealMe pipeline: limited captures, auxiliary data construction, video diffusion enhancement, and optimized 4D reconstruction

Model Architecture

ControlNet and diffusion transformer architecture with layout-aware positional embeddings and factorized spatio-temporal attention

ControlNet. A frozen VAE encodes the auxiliary videos into condition latents, while a frozen text encoder provides prompt features. ControlNet converts both into control signals injected as conditioning residuals into each DiT block.
Layout-aware positional embeddings. We extend image positional IDs with frame and viewpoint indices alongside the two latent-grid coordinates. Rotary embeddings over these four axes distinguish tokens at the same grid location across different frames and views.
DiT with factorized spatio-temporal attention. DiT blocks replace full attention with separate temporal and spatial patterns, where the spatial axis indexes viewpoints rather than image patches. This propagates information across frames and views during denoising, improving 4D consistency.

Comparisons

RealMe outperforms state-of-the-art methods in novel-view synthesis quality and 4D consistency on both datasets.

FTGS

DIFIX3D+

Diffuman4D

Ours

Detail comparison

Indoor capture

Zoom in for a better view

Indoor portrait from the low-resolution input — Input

Indoor portrait enhanced by FlashVSR — FlashVSR

Indoor portrait enhanced by RealMe — Ours

Studio capture

Studio portrait from the low-resolution input — Input

Studio portrait enhanced by FlashVSR — FlashVSR

Studio portrait enhanced by RealMe — Ours

Compared with STAR and FlashVSR, RealMe restores finer facial, hair, and clothing detail with a more natural appearance.

RealVV Dataset

Our self-captured RealVV dataset includes varied actors, actions, scenes, camera baselines, and focal settings.

Short-focus

Long-focus

Mask

Camera rigs

Scene 1: seated performer, short-focus view — Short-focus

Scene 1: seated performer, long-focus view — Long-focus

Scene 2: performer in a dark studio, short-focus view — Short-focus

Scene 2: performer in a dark studio, long-focus view — Long-focus

Scene 3: performer in a purple-lit studio, short-focus view — Short-focus

Scene 3: performer in a purple-lit studio, long-focus view — Long-focus

Scene 4: performer in a green studio, short-focus view — Short-focus

Scene 4: performer in a green studio, long-focus view — Long-focus

Scene 5: performer in a white studio, short-focus view — Short-focus

Scene 5: performer in a white studio, long-focus view — Long-focus

Acknowledgements

We would like to thank Yujie Guo, Zhen Xu, and Taoming Liu for their insightful discussions.

This work was partially supported by the National Key R&D Program of China (No. 2024YFB2809105), the Zhejiang Provincial Natural Science Foundation of China (No. LR25F020003), and the Information Technology Center and State Key Lab of CAD&CG, Zhejiang University.

All actors shown in the dataset have authorized the use of their footage.

Citation

@inproceedings{zhou2026realme,
  title     = {RealMe: Immersive Volumetric Video from Minimal Captures},
  author    = {Zhou, Chenxu and Yu, Zhiyuan and Jin, Yudong and Shentu, Yinji and Zhang, Yu and Sun, Jiaming and Zhou, Xiaowei},
  booktitle = {SIGGRAPH Asia 2026 Conference Papers},
  year      = {2026},
  doi       = {10.1145/3829340.3842349}
}