BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion (CVPR 2023)

Michael J. Black, Priyanka Patel, Joachim Tesch, Jinlong Yang

BEDLAM is a synthetic dataset consisting of realistic monocular RGB videos with accurately simulated 3D bodies, diverse clothing, and varied environments, enabling state-of-the-art accuracy in 3D human pose and shape estimation tasks using synthetic training data. As part of the project, my work involves training various regressors to estimate human pose and shape using the BEDLAM dataset. Additionally, I conducted comprehensive ablations and evaluations on different benchmarks.

AGORA: Avatars in Geography Optimized for Regression Analysis (CVPR 2021)

Priyanka Patel, Chun-Hao P. Huang, Joachim Tesch, David T. Hoffmann, Shashank Tripathi and Michael J. Black

AGORA is a synthetic dataset that addresses the domain gap in 3D human pose and shape estimation dataset by providing highly realistic images with accurate ground truth in SMPL/SMPL-X format, including diverse poses, natural clothing, and variations in lighting and environments. It exposes the limitations of current methods and enables the development of improved models.

COMA in Pytorch

COMA (Convolutional Mesh Autoencoder) is an advanced technique for accurately representing 3D faces in a low-dimensional latent space. It surpasses the performance of linear PCA models while employing fewer parameters. In order to make it more accessible and adaptable, I have successfully ported the original code from Tensorflow to PyTorch, leveraging the capabilities of the torch-geometric library.

Model Registration Pipeline for 3D Scans

A modular and scalable pipeline for registering different body models, such as SMPL, SMPL-X, and SMIL, to 3D scans in Pytorch. The key features include rendering 3D scans from multiple camera viewpoints, estimating 2D keypoints in each view, optimizing model poses by minimizing the projected 2D keypoints loss across all views, and utilizing point-to-surface distance for precise alignment of the model and the scan after an initial proximity estimation.

Registration for clothed 3D Scans

As part of the project AGORA, I work on registering SMPL-X model to 3D scans with clothing and hair. This task presents a greater challenge compared to scans with minimal clothing, as the point-to-surface distance tends to distort the body in order to accommodate the clothing. To address this issue, I employ Graphonomy, which help in separating the scan into skin and clothing vertices. I further implemented an optimization term to minimize the point to surface distance for skin region while simultaneously ensuring the body remains appropriately enclosed within the clothing.