Controllable and Interactive 3D Assets Generation with Proxy-Guided Conditioning


  • 1Zhejiang University
  • 2ByteDance


We present a novel controllable and interactive 3D assets modeling framework, named Coin3D. Coin3D allows users to control the 3D generation using a coarse geometry proxy assembled from basic shapes, and introduces an interactive generation workflow to support seamless local part editing while delivering responsive 3D object previewing within a few seconds. To this end, we develop several techniques, including the 3D adapter that applies volumetric coarse shape control to the diffusion model, proxy-bounded editing strategy for precise part editing, progressive volume cache to support responsive preview, and volume-SDS to ensure consistent mesh reconstruction. Extensive experiments of interactive generation and editing on diverse shape proxies demonstrate that our method achieves superior controllability and flexibility in the 3D assets generation task.



Description of the image

Given a coarse shape proxy and user prompts that describe the identity, our method first constructs 2D image candidates from the proxy's silhouette and 3D proxy samples as input conditions. Then, we employ a 3D adapter to integrate 3D-aware control to the diffusion's denoising process with a 3D control volume, yielding multiview images of the object. By fully leveraging the 3D control volume, we realize accelerated 3D previewing with volume cache and improve mesh reconstruction quality.

3D-Aware Conditioned Generation

We use a novel 3D adapter module that seamlessly integrates proxy-guided controls into the diffusion pipeline.

Proxy-Bounded Part Editing

We design an interactive local part editing workflow through the proxy's elements. Specifically, users can annotate specific part of the proxy and regenerate the 3D content by updating the 2D conditions and 3D volumes.

Interactive Generation Workflow

We demonstrate interactive workflow by deploying our method into Blender (3D modeling software) and ComfyUI (web-based generation interface).

More Examples

We show more examples of controllable 3D generation, where we use coarse proxies as conditions to generate fine-grained textured objects.

Part-Editing Examples

We show more examples of interactive part editing, where we can precisely edit/regenerate the desired part via the annotated proxy pieces (e.g., adding torch, candles, or changing the pumpkin to an apple).


title ={Coin3D: Controllable and Interactive 3D Assets Generation with Proxy-Guided Conditioning}, author ={Dong, Wenqi and Yang, Bangbang and Ma, Lin and Liu, Xiao and Cui, Liyuan and Bao, Hujun and Ma, Yuewen and Cui, Zhaopeng}, year ={2024}, eprint ={2405.08054}, archivePrefix ={arXiv}, primaryClass ={cs.GR} }

Misc & Acknowledgements

Wenqi Dong conducted this work during his internship at PICO, ByteDance. The authors from Zhejiang University are also affiliated with the State Key Laboratory of CAD&CG. The webpage template is borrowed from and