CharacterGen: Efficient 3D Character Generation from Single Images

with Multi-View Pose Calibration

SIGGRAPH(TOG) 2024
1Tsinghua University, 2VAST

Corresponding Author
teaser image CharacterGen takes a single input image and generates 3D pose-unified character meshes with high-quality and consistent appearance, which can be directly utilized in downstream rigging and animation workflows.

Abstract

In this paper, we present CharacterGen, a framework developed to efficiently generate 3D characters. CharacterGen introduces a streamlined generation pipeline along with an image-conditioned multi-view diffusion model. This model effectively calibrates input poses to a canonical form while retaining key attributes of the input image, thereby addressing the challenges posed by diverse poses. A transformer-based, generalizable sparse-view reconstruction model is the other core component of our approach, facilitating the creation of detailed 3D models from multi-view images. We also adopt a texture-back-projection strategy to produce high-quality texture map. Additionally, we have curated a dataset of anime characters, rendered in multiple poses and views, to train and evaluate our model. Our approach has been thoroughly evaluated through quantitative and qualitative experiments, showing its proficiency in generating 3D characters with high-quality shapes and textures, ready for downstream applications such as rigging and animation. Our code, dataset and pretrained weight will be available soon.

pipeline image

We show the whole pipeline above of how we generate four views of consistent images.

lrm image

We show the pipeline above on how we generate final refined character meshes from generated multi-view images. In the first stage, we utilize a deep transformer-based network to generate an character with a coarse texture and then adopt our texture back-projection strategy to enhance the appearance of the generated mesh.

Acknowledgements

Thanks very much to many friends for their unselfish help with our work. I'm extremely grateful to Yuanchen, Yangguang, and Yuan Liang for their guidance on code details and ideas, as well as to uncle-lv and 33 Sangyu for their assistance with this website setup and JavaScript render scripts!

Our Results

Single Input Image

2D Multi-View Images

3D Character

1
2
2
2
3
3

2D Comparsion Results

res 2d image

3D Comparsion Results

Input Image

CharacterGen

ImageDream

Magic123