Human-Robot-Scene Interaction and Collaboration

Introduction

Intelligent robots are advancing rapidly, with embodied agents increasingly expected to work and live alongside humans in households, factories, hospitals, schools, etc. For these agents to operate safely, socially, and intelligently, they must effectively interact with humans and adapt to changing environments. Moreover, such interactions can transform human behavior and even reshape the environment—for example, through adjustments in human motion during robot-assisted handovers or the redesign of objects for improved robotic grasping. Beyond established research in human-human and human-scene interactions, vast opportunities remain in exploring human-robot-scene collaboration. This workshop will explore the integration of embodied agents into dynamic human-robot-scene interactions. Our focus is on, but not limited to:

• Transferring knowledge from human-human and human-scene interaction and collaboration to inform the development of humanoids and other embodied agents (e.g., via retargeting).
• Exploring different methods for deriving visual representations that capture object properties, dynamics, and affordances relevant to human-robot collaboration.
• Investigating methods for modeling and predicting human intentions to enable robots to anticipate actions and respond safely.
• Integrating robots into interactive settings to foster seamless and effective teamwork.
• Establishing meaningful benchmarks and metrics to measure advancements in human-robot-scene interaction and collaboration.

Papers

Oral

• RHINO: Learning Real-Time Humanoid-Human-Object Interaction from Human Demonstrations
• DialNav: Multi-turn Dialog Navigation with a Remote Guide
• PICO: Reconstructing 3D People In Contact with Objects
• Integrating LMM Planners and 3D Skill Policies for Generalizable Manipulation

Poster

• ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making
• Communicative PARTNR: Natural Language Communication under Partial Observability in Human-Robot Collaboration
• Radar-Conditioned 3D Bounding Box Diffusion for Indoor Human Perception
• HiERO: understanding the hierarchy of human behavior enhances reasoning on egocentric videos
• FOREcasting human activities via latent SCENE graphs diffusion
• Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions

Challenges

We are excited to announce the Multi-Terrain Humanoid Locomotion Challenge and Humanoid-Object Interaction Challenge, which will be held in conjunction with the workshop. The challenge aims to foster advancements in humanoid-scene interaction by providing a platform for researchers to showcase their work on embodied agents in dynamic environments. For more details, please visit the challenge websites.

Multi-Terrain Humanoid Locomotion Challenge

🏆 Awards: 🥇 First Prize ($1000) 🥈 Second Prize ($500) 🥉 Third Prize ($300)

Humanoid-Object Interaction Challenge

🏆 Awards: 🥇 First Prize ($1000) 🥈 Second Prize ($500) 🥉 Third Prize ($300)

Schedule

Invited Speakers

listed alphabetically

Jiayuan Mao

MIT

Roozbeh Mottaghi

Meta AI

Mahi Shafiullah

BAIR and FAIR

Xianghui Xie

University of Tübingen

Hang Zhao

Tsinghua University

Organizers

Jiayuan Gu

ShanghaiTech University

Xingyu Lin

OpenAI

Fangchen Liu

UC Berkeley

Yuexin Ma

ShanghaiTech University

Martin Magnusson

Örebro University

Sören Schwertfeger

ShanghaiTech University

Ye Shi

ShanghaiTech University

Hao Su

UC San Diego

Jingya Wang

ShanghaiTech University

Lan Xu

ShanghaiTech University

Li Yi

Tsinghua University

Program Committee