Abstract

We present a novel controllable and interactive 3D assets modeling framework, named Coin3D. Coin3D allows users to control the 3D generation using a coarse geometry proxy assembled from basic shapes, and introduces an interactive generation workflow to support seamless local part editing while delivering responsive 3D object previewing within a few seconds. To this end, we develop several techniques, including the 3D adapter that applies volumetric coarse shape control to the diffusion model, proxy-bounded editing strategy for precise part editing, progressive volume cache to support responsive preview, and volume-SDS to ensure consistent mesh reconstruction. Extensive experiments of interactive generation and editing on diverse shape proxies demonstrate that our method achieves superior controllability and flexibility in the 3D assets generation task.

Video

Pipeline

Given a coarse shape proxy and user prompts that describe the identity, our method first constructs 2D image candidates from the proxy's silhouette and 3D proxy samples as input conditions. Then, we employ a 3D adapter to integrate 3D-aware control to the diffusion's denoising process with a 3D control volume, yielding multiview images of the object. By fully leveraging the 3D control volume, we realize accelerated 3D previewing with volume cache and improve mesh reconstruction quality.

3D-Aware Conditioned Generation

We use a novel 3D adapter module that seamlessly integrates proxy-guided controls into the diffusion pipeline.

Proxy-Bounded Part Editing

We design an interactive local part editing workflow through the proxy's elements. Specifically, users can annotate specific part of the proxy and regenerate the 3D content by updating the 2D conditions and 3D volumes.

Interactive Generation Workflow

We demonstrate interactive workflow by deploying our method into Blender (3D modeling software) and ComfyUI (web-based generation interface).

More Examples

Part-Editing Examples