Research Overview

Humans build generalizable and explainable representations of their environment through interaction, observation, imitation, intervention, and language. The following research produced from a collaborative, team-based lab explores how artificial agents can use these five concepts to learn robust and transferable representations of tasks and environments.

Representation learning: The role of language in building generalizable representations

Center for Vision, Cognition, Learning, and Autonomy, UCLA• Mar 2018 - Present

• Created a virtual playground for embodied AI to learn interpretable, common-sense representations of its environment.
• Built a simulated environment using Unreal Engine 4 (UE4) that couples language and vision in a scene graph.
• Devised a dataset consisting of images, language labels, and object segmentation.
• Ongoing: Build representation learning algorithm to use language labels to decompose latent encoding of the environment.

Collaborators:

• Xu Xie, University of California, Los Angeles
• John Dang, University of California, Los Angeles
• Samuel Hunter, University of California, Los Angeles
• Paymon Haddad, University of California, Los Angeles
• Yixin Zhu, University of California, Los Angeles
• Song-Chun Zhu, University of California, Los Angeles

Explainable AI: How can robots explain their behavior to foster trust from humans?

Center for Vision, Cognition, Learning, and Autonomy, UCLA• Feb 2018 - Dec 2019

• Expanded prior imitation learning work by building human-understandable visual interfaces to describe the robot's haptic network and And-Or Graph.
• Showed that robots may need to "think" one way to complete a task but another way to explain their behavior: the explanations that best fostered trust were not the model components that best aided the robot to achieve the task.
• Result: Consider explainability as a first-class citizen when building AI systems that interact with humans.

Collaborators:

• Feng Gao, University of California, Los Angeles
• Hangxin Liu, University of California, Los Angeles
• Xu Xie, University of California, Los Angeles
• Siyuan Qi, University of California, Los Angeles
• Brandon Rothrock, NASA Jet Propulsion Laboratory
• Yixin Zhu, University of California, Los Angeles
• Ying Nian Wu, University of California, Los Angeles
• Hongjing Lu, University of California, Los Angeles
• Song-Chun Zhu, University of California, Los Angeles

Causal learning: Virtual escape room to examine how humans and AI learn transferable causal representations

Center for Vision, Cognition, Learning, and Autonomy, UCLA• Feb 2017 - Feb 2020

• Built virtual "escape room" to test causal generalization; surface-level features change room to room while each room is governed by a common abstract causal structure (series of levers) that describes the required actions to "unlock" the room.
• Ran human subject experiments to verify human learners are capable of discovering the correct abstract causal structure.
• Built hierarchical Bayesian model to achieve similar performance as human learners. This causal model was able to solve the escape room while seven state-of-the-art model-free reinforcement learning algorithms failed at the task.
• Result: Both structural abstraction and feature generalization are critical for transfer learning and generalization.

Collaborators:

• Yixin Zhu, University of California, Los Angeles
• James Kubricht, University of California, Los Angeles
• Brandon Rothrock, NASA Jet Propulsion Laboratory
• Hongjing Lu, University of California, Los Angeles
• Song-Chun Zhu, University of California, Los Angeles

Imitation learning: Training a robot to twist open a medicine bottle

Center for Vision, Cognition, Learning, and Autonomy, UCLA• December 2015 - Feb 2017

• Captured the complex human hand forces required to open seven different medicine bottles with a tactical glove covered in IMUs (inertial measurement units) and force sensors.
• Constructed robot action planner using a haptic network, And-Or Graph, and the generalized Earley parser.
• Result: Questioned the common structure humans see in the procedure to open any medicine bottle vs. the widely varying forces and action sequences used by the robot for each bottle. This prompted an investigation of abstraction and generalization...

Collaborators:

• Feng Gao, University of California, Los Angeles
• Hangxin Liu, University of California, Los Angeles
• Xu Xie, University of California, Los Angeles
• Matt Millar, University of California, Los Angeles
• Siyuan Qi, University of California, Los Angeles
• Yixin Zhu, University of California, Los Angeles
• Brandon Rothrock, NASA Jet Propulsion Laboratory
• Song-Chun Zhu, University of California, Los Angeles

1. Brain Machine Interface Using Emotiv EPOC to Control Robai Cyton Robotic Arm