A collection of my human-related work, including HCMoCo, Garment4D, AvatarCLIP, EVA3D, EgoLM.

EgoLM is a language model-based framework that tracks and understands egocentric motions from multi-modal inputs, i.e., egocentric videos and sparse motion sensors.

Text-to-3D Generation within 5 Minutes! A two-stage design, utilizing both 3D difffusion prior and 2D priors.

Extension of the CVPR21 Version; Extend DS-Net to 4D panoptic LiDAR segmentation by the temporally unified instance clustering on aligned LiDAR frames.

Reconstruct human NeRF from a single image in one forward pass!

EVA3D is a high-quality unconditional 3D human generative model that only requires 2D image collections for training.

AvatarCLIP empowers layman users to customize a 3D avatar with the desired shape and texture, and drive the avatar with the described motions using solely natural languages.

The first to leverage the multi-modal nature of human data (e.g. RGB, depth, 2D key-points) for effective human-centric representation learning.

The first attempt at separable and interpretable garment reconstruction from point cloud sequences, especially challenging loose garments.

Rank 1st in the public leaderboard of SemanticKITTI panoptic segmentation (2020-11-16); A learnable clustering module is designed to adapt kernel functions to complex point distributions.

GeneMAN is a generalizable framework for single-view-to-3D human reconstruction, built on a collection of multi-source human data.

I contribute two chapters in this book: Text-Conditioned Zero-Shot 3D Avatar Creation and Animation, and Text-Driven 3D Human Motion Generation.

CityDreamer4D is a framework for unbounded 4D city generation that decouples static and dynamic scenes, achieving superior realism, multi-view consistency, and diverse styles.

A collection of my human-related work, including HCMoCo, Garment4D, AvatarCLIP, EVA3D, EgoLM.

We generate 3D digital humans using 3D diffusion model in a controllable, layer-wise way.

EgoLM is a language model-based framework that tracks and understands egocentric motions from multi-modal inputs, i.e., egocentric videos and sparse motion sensors.

With mesh attention designed for efficient cross-view feature fusion, MEAT is the first human multiview diffusion model that can generate dense, view-consistent multiview images at a resolution of 1024x1024.

3DTopia-XL scales high-quality 3D asset generation using Diffusion Transformer (DiT) built upon an expressive and efficient 3D representation, PrimX. The denoising process takes 5 seconds to generate a 3D PBR asset from text / image input which is ready for graphics pipeline to use.

GaussianCity is a framework for efficient unbounded 3D city generation using 3D Gaussian Splatting.

We introduce EgoLife, a project to develop an egocentric life assistant that accompanies and enhances personal efficiency through AI-powered wearable glasses.

GaussianAnything generates high-quality and editable surfel Gaussians through a cascaded 3D diffusion pipeline, given single-view images or texts as the conditions.

Extension of our ICLR 2024 paper DiffTF. Joint training of diffusion model and Triplane representation increases the generation quality.

We propose HMD², the first system for the online generation of full-body self-motion using a single head-mounted device (e.g. Project Aria Glasses) equipped with an outward-facing camera in complex and diverse environments.

A large-scale, diverse, richly annotated human motion dataset collected in the wild with multi-modal egocentric devices.

LN3Diff creates high-quality 3D object mesh from text within 8 V100-SECONDS.

StructLDM is a diffusion-based unconditional 3D human generative model learned from 2D images.

Large Motion Model (LMM) is a motion-centric, multi-modal framework that unifies mainstream motion generation tasks into a generalist model.

Dynamic human rendering with the joint modeling of motion dynamics and appearance.

Unbouned 3D cities generated from 2D image collections!

DiffTF achieves state-of-the-art large-vocabulary 3D object generation performance with 3D-aware transformers.

The first diffusion-model-based text-driven motion generation framework with probabilistic mapping, realistic synthesis and multi-level manipulation ability.

Extension of the CVPR21 Version; Extend DS-Net to 4D panoptic LiDAR segmentation by the temporally unified instance clustering on aligned LiDAR frames.

PrimDiffusion performs the diffusion and denoising process on a set of primitives which compactly represent 3D humans.

To allow artificial intelligence to develop a comprehensive understanding of a 4D world, we introduce 4D Panoptic Scene Graph (PSG-4D), a new representation that bridges the raw visual data perceived in a dynamic 4D world and high-level visual understanding.

Reconstruct human NeRF from a single image in one forward pass!

We learn a style field that deforms real 3D faces to styleized 3D faces.

ReMoDiffuse is a diffusion-model-based motion generation framework that integrates a retrieval mechanism to refine the denoising process, which enhances the generalizability and diversity.

EVA3D is a high-quality unconditional 3D human generative model that only requires 2D image collections for training.

A large-scale multi-modal (color images, point clouds, keypoints, SMPL parameters, and textured meshes) 4D human dataset with 1000 human subjects, 400k sequences and 60M frames.

AvatarCLIP empowers layman users to customize a 3D avatar with the desired shape and texture, and drive the avatar with the described motions using solely natural languages.

The first to leverage the multi-modal nature of human data (e.g. RGB, depth, 2D key-points) for effective human-centric representation learning.

The first attempt at separable and interpretable garment reconstruction from point cloud sequences, especially challenging loose garments.

Rank 1st in the public leaderboard of SemanticKITTI panoptic segmentation (2020-11-16); A learnable clustering module is designed to adapt kernel functions to complex point distributions.

Journal Extension of the CVPR21 version; Extend the cylindrical convolution to more general LiDAR-based perception tasks.

Rank 1st in the public leaderboard of SemanticKITTI semantic segmentation (2020-11-16); Cylindrical 3D convolution is designed to explore the 3D geometric pattern of LiDAR point clouds.

To learn discriminative features on point clouds by encoding the fine-grained contexts inside and among local regions simultaneously.

We propose EgoTwin, a diffusion-based framework that jointly generates egocentric video and human motion in a viewpoint consistent and causally coherent manner. Generated videos can be lifted into 3D scenes using camera poses derived from human motion via 3D Gaussian Splatting.

4DNeX is a feed-forward framework for generating 4D scene representations from a single image by fine-tuning a video diffusion model. It produces high-quality dynamic point clouds and enables downstream tasks such as novel-view video synthesis with strong generalizability.

STream3R reformulates dense 3D reconstruction into a sequential registration task with causal attention.

This survey provides a systematic overview of state-of-the-art approaches, organizing them into four paradigms: procedural generation, neural 3D-based generation, image-based generation, and video-based generation.

We present a new perspective that organizes existing dynamic reconstruction methods into five progressive levels of 4D spatial intelligence.

FashionEngine is an interactive 3D human generation and editing system with multimodal control (e.g., texts, images, hand-drawing sketches).

Text-to-3D Generation within 5 Minutes! A two-stage design, utilizing both 3D difffusion prior and 2D priors.

SMPL reconstruction from real depth sensor, which are partial point cloud inputs.

3DTopia-XL scales high-quality 3D asset generation using Diffusion Transformer (DiT) built upon an expressive and efficient 3D representation, PrimX. The denoising process takes 5 seconds to generate a 3D PBR asset from text / image input which is ready for graphics pipeline to use.

GaussianAnything generates high-quality and editable surfel Gaussians through a cascaded 3D diffusion pipeline, given single-view images or texts as the conditions.

Extension of our ICLR 2024 paper DiffTF. Joint training of diffusion model and Triplane representation increases the generation quality.

Text-to-3D Generation within 5 Minutes! A two-stage design, utilizing both 3D difffusion prior and 2D priors.

LN3Diff creates high-quality 3D object mesh from text within 8 V100-SECONDS.

DiffTF achieves state-of-the-art large-vocabulary 3D object generation performance with 3D-aware transformers.

FashionEngine is an interactive 3D human generation and editing system with multimodal control (e.g., texts, images, hand-drawing sketches).

StructLDM is a diffusion-based unconditional 3D human generative model learned from 2D images.

PrimDiffusion performs the diffusion and denoising process on a set of primitives which compactly represent 3D humans.

We learn a style field that deforms real 3D faces to styleized 3D faces.

EVA3D is a high-quality unconditional 3D human generative model that only requires 2D image collections for training.

AvatarCLIP empowers layman users to customize a 3D avatar with the desired shape and texture, and drive the avatar with the described motions using solely natural languages.

We generate 3D digital humans using 3D diffusion model in a controllable, layer-wise way.

4DNeX is a feed-forward framework for generating 4D scene representations from a single image by fine-tuning a video diffusion model. It produces high-quality dynamic point clouds and enables downstream tasks such as novel-view video synthesis with strong generalizability.

CityDreamer4D is a framework for unbounded 4D city generation that decouples static and dynamic scenes, achieving superior realism, multi-view consistency, and diverse styles.

GaussianCity is a framework for efficient unbounded 3D city generation using 3D Gaussian Splatting.

Unbouned 3D cities generated from 2D image collections!

EgoLM is a language model-based framework that tracks and understands egocentric motions from multi-modal inputs, i.e., egocentric videos and sparse motion sensors.

We propose HMD², the first system for the online generation of full-body self-motion using a single head-mounted device (e.g. Project Aria Glasses) equipped with an outward-facing camera in complex and diverse environments.

A large-scale, diverse, richly annotated human motion dataset collected in the wild with multi-modal egocentric devices.

Large Motion Model (LMM) is a motion-centric, multi-modal framework that unifies mainstream motion generation tasks into a generalist model.

The first diffusion-model-based text-driven motion generation framework with probabilistic mapping, realistic synthesis and multi-level manipulation ability.

ReMoDiffuse is a diffusion-model-based motion generation framework that integrates a retrieval mechanism to refine the denoising process, which enhances the generalizability and diversity.

A large-scale multi-modal (color images, point clouds, keypoints, SMPL parameters, and textured meshes) 4D human dataset with 1000 human subjects, 400k sequences and 60M frames.

AvatarCLIP empowers layman users to customize a 3D avatar with the desired shape and texture, and drive the avatar with the described motions using solely natural languages.

SMPL reconstruction from real depth sensor, which are partial point cloud inputs.

STream3R reformulates dense 3D reconstruction into a sequential registration task with causal attention.

With mesh attention designed for efficient cross-view feature fusion, MEAT is the first human multiview diffusion model that can generate dense, view-consistent multiview images at a resolution of 1024x1024.

GeneMAN is a generalizable framework for single-view-to-3D human reconstruction, built on a collection of multi-source human data.

Dynamic human rendering with the joint modeling of motion dynamics and appearance.

Reconstruct human NeRF from a single image in one forward pass!

The first attempt at separable and interpretable garment reconstruction from point cloud sequences, especially challenging loose garments.

EgoLM is a language model-based framework that tracks and understands egocentric motions from multi-modal inputs, i.e., egocentric videos and sparse motion sensors.

Extension of the CVPR21 Version; Extend DS-Net to 4D panoptic LiDAR segmentation by the temporally unified instance clustering on aligned LiDAR frames.

To allow artificial intelligence to develop a comprehensive understanding of a 4D world, we introduce 4D Panoptic Scene Graph (PSG-4D), a new representation that bridges the raw visual data perceived in a dynamic 4D world and high-level visual understanding.

The first to leverage the multi-modal nature of human data (e.g. RGB, depth, 2D key-points) for effective human-centric representation learning.

Rank 1st in the public leaderboard of SemanticKITTI panoptic segmentation (2020-11-16); A learnable clustering module is designed to adapt kernel functions to complex point distributions.

Journal Extension of the CVPR21 version; Extend the cylindrical convolution to more general LiDAR-based perception tasks.

Rank 1st in the public leaderboard of SemanticKITTI semantic segmentation (2020-11-16); Cylindrical 3D convolution is designed to explore the 3D geometric pattern of LiDAR point clouds.

To learn discriminative features on point clouds by encoding the fine-grained contexts inside and among local regions simultaneously.

We propose EgoTwin, a diffusion-based framework that jointly generates egocentric video and human motion in a viewpoint consistent and causally coherent manner. Generated videos can be lifted into 3D scenes using camera poses derived from human motion via 3D Gaussian Splatting.

EgoLM is a language model-based framework that tracks and understands egocentric motions from multi-modal inputs, i.e., egocentric videos and sparse motion sensors.

We introduce EgoLife, a project to develop an egocentric life assistant that accompanies and enhances personal efficiency through AI-powered wearable glasses.

We propose HMD², the first system for the online generation of full-body self-motion using a single head-mounted device (e.g. Project Aria Glasses) equipped with an outward-facing camera in complex and diverse environments.

A large-scale, diverse, richly annotated human motion dataset collected in the wild with multi-modal egocentric devices.