InstaScene Logo Towards Complete 3D Instance Decomposition and Reconstruction from Cluttered Scenes

ICCV 2025

Zesong Yang1,2, Bangbang Yang2, Wenqi Dong1, Chenxuan Cao1, Liyuan Cui1,
Yuewen Ma2, Zhaopeng Cui1, Hujun Bao†1,
1State Key Lab of CAD & CG, Zhejiang University, 2ByteDance
†Corresponding Authors
Paper ArXiv Supplementary Code(Coming Soon) Video

TL;DR: InstaScene integrates spatial contrastive learning for precise instance-aware decomposition and in-situ generative priors for complete 3D reconstruction from cluttered and partially observed complex scenes.


InstaScene allows users to pick up and decompose arbitrary instances from cluttered environments, while automatically reconstructing them into complete objects with intact geometry and appearance that align with the physical world.


Abstract


Humans can naturally identify and mentally complete occluded objects in cluttered environments. However, imparting similar cognitive ability to robotics remains challenging even with advanced reconstruction techniques, which models scenes as undifferentiated wholes and fails to recognize complete object from partial observations. In this paper, we propose InstaScene, a new paradigm towards holistic 3D perception of complex scenes with a primary goal: decomposing arbitrary instances while ensuring complete reconstruction. To achieve precise decomposition, we develop a novel spatial contrastive learning by tracing rasterization of each instance across views, significantly enhancing semantic supervision in cluttered scenes. To overcome incompleteness from limited observations, we introduce in-situ generation that harnesses valuable observations and geometric cues, effectively guiding 3D generative models to reconstruct complete instances that seamlessly align with the real world. Experiments on scene decomposition and object completion across complex real-world and synthetic scenes demonstrate that our method achieves superior decomposition accuracy while producing geometrically faithful and visually intact objects.

Pipeline



Given a reconstructed Gaussian Splatting scene, our method first clusters and filters 2D segmentation masks by tracing the rasterization of Gaussian Splatting, which yields 2D and 3D instance masks. Then, we use spatial contrastive learning with mask supervision to train a feature field that achieves high-quality scene decomposition. Finally, for each decomposed incomplete object, we conduct an in-situ generation pipeline that takes all known observations and geometric cues as omni-conditions to the 3D generative model to obtain supplemented views, which will be jointly fine-tuned with source training views to obtain a complete object.

Fine-grained Instance-level Scene Decomposition

Examples from ZipNeRF and LERF dataset show that the baselines yield highly noisy segmentation results in such complex scenes, while our method achieves fine-grained instance-level segmentation
GSGroup Feat. GSGroup Mask LangSplat Feat. LangSplat Mask Ours Feat. Ours Mask

Fine-Grained & Interactive Decomposition

Our method learns a highly distinguishable feature field, enabling fine-grained and real-time interactive segmentation.

In-situ Generation with Generative Diffusion Prior

We propose a novel in-situ generation pipeline, which tames the generic 3D generative model with all known information (e.g., partial observations and incomplete reconstructed geometry) to achieve realistic and complete instance reconstruction even with occlusions and limited observations.

In-situ Generation VS. Generic Image-to-3D

We put the reconstructed instance back into the original scene. Compared to generic image-to-3D methods, our in-situ generation ensures both appearance and geometric consistency with the original scene while faithfully completing the unseen regions.

More Results & Interactive Manipulation

We present the application of interactive manipulation on instances reconstructed using our in-situ generation method, including object duplication, translation, and rotation. The naive extraction of instances without in-situ generation results in significant incompleteness, and hinders the application in simulated environments.

BibTeX

@inproceedings{yang2025instascene,
    title={InstaScene: Towards Complete 3D Instance Decomposition and Reconstruction from Cluttered Scenes},
    author={Yang, Zesong and Yang, Bangbang and Dong, Wenqi and Cao, Chenxuan and Cui, Liyuan and Ma, Yuewen and Cui, Zhaopeng and Bao, Hujun},
    booktitle=ICCV,
    year={2025}
}