The third Workshop on
Reconstruction of Human-Object Interactions (RHOBIN)

Understanding Human Motion in the Wild

Machine perception of human motion is crucial in many areas, from Robotics to Mixed Reality. However, developing perception algorithms that work in the wild is challenging, due to the complexity of human behavior and the difficulty of acquiring 3D data at scale. In this talk, we will focus on how to tackle these challenges looking at recent advances in computer vision and machine learning. First, we will look at robust motion reconstruction algorithms that, taking just monocular RGB(-D) videos as input, can accurately capture human motion even in the presence of noise and occlusions. Second, we will discuss the importance of capturing diverse human motion data at scale, introducing the Nymeria dataset. Nymeria is currently the largest multi-modal dataset for in-the-wild human motion, providing both ground-truth 3D pose annotations and motion-language descriptions. Finally, we will show how we can leverage this data to effectively reason about human motion, connecting it to natural language. We will present EgoLM, a recent multi-modal framework that leverages Large Language Models to simultaneously track and understand human motion in egocentric scenarios.

From Hands to Feet: Contact-Driven Modelling of Human-Object-World Interactions

Understanding how humans interact with objects and the physical world is fundamental to modeling everyday behavior in 3D. This talk focuses on contacts, between hands and objects, and between the feet and the world, as driving cues for 3D reconstruction and generation. First, we will look at natural language-based generation of hand-object contacts, to improve hand-object reconstruction and generation. Language provides a natural way of specifying I will then introduce our state-of-the-art approach on world-coordinate human mesh recovery based on inferred contacts such as the feet.