| CARVIEW |
Jacky Liang
Research Scientist Robotics at Google
PhD from Robotics Institute
Carnegie Mellon University
Blog |
CV
Hello!
I’m a research scientist at Google DeepMind, and my research focus is on foundation models for robotics. I received my PhD from Carnegie Mellon University’s Robotics Institute, co-advised by Prof. Oliver Kroemer and Prof. Maxim Likhachev. My PhD thesis is on Learning with Structured Priors for Robust Robot Manipulation. A recording of my PhD defense is here.
Previously, I interned at NVIDIA for the Isaac project, applying GPU-accelerated physics simulations for robot learning. During my time at UC Berkeley, I did research in robot grasping and manipulation with Prof. Ken Goldberg.
In my spare time I write for and co-run Skynet Today and the Last Week in AI newsletter. Our goal is to demystify and disseminate AI news without hype. Check us out and subscribe if you like what you see :)
Research
Conference Papers
Code as Policies: Language Model Programs for Embodied Control
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, Andy Zeng. International Conference on Robotics and Automation (ICRA). June 2023. Outstanding Paper in Robot Learning. [arXiv] [Website]
Learning Preconditions of Hybrid Force-Velocity Controllers for Contact-Rich Manipulation
Jacky Liang, Xianyi Cheng, Oliver Kroemer. Conference on Robot Learning (CoRL). Dec 2022. [arXiv] [Website]
Inner Monologue: Embodied Reasoning through Planning with Language Models
Wenlong Huang*, Fei Xia*, Ted Xiao*, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, Brian Ichter. Conference on Robot Learning (CoRL). Dec 2022. [arXiv] [Website]
Search-Based Task Planning with Learned Skill Effect Models for Lifelong Robotic Manipulation
Jacky Liang*, Mohit Sharma*, Alex LaGrassa, Shivam Vats, Saumya Saxena, Oliver Kroemer.
International Conference on Robotics and Automation (ICRA).
May 2022.
[arXiv]
[Website]
Visual Identification of Articulated Object Parts
Vicky Zeng, Timothy E. Lee*, Jacky Liang*, Oliver Kroemer.
International Conference on Intelligent Robots and Systems (IROS).
September 2021.
[arXiv]
[Website]
Contact Localization for Robot Arms in Motion without Torque Sensing
Jacky Liang, Oliver Kroemer.
International Conference on Robotics and Automation (ICRA).
May 2021.
[arXiv]
[Website]
Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulation
Mohit Sharma*, Jacky Liang*, Jialiang Zhao, Alex LaGrassa, Oliver Kroemer.
Conference on Robot Learning (CoRL).
November 2020.
Plenary Presentation.
[arXiv]
[Website]
Learning Active Task-Oriented Exploration Policies for Bridging the Sim-to-Real Gap
Jacky Liang, Saumya Saxena, Oliver Kroemer.
Robotics: Science and Systems (RSS).
July 2020.
[arXiv]
[Website]
In-Hand Object Pose Tracking via Contact Feedback and GPU-Accelerated Robotic Simulation
Jacky Liang, Ankur Handa, Karl Van Wyk, Viktor Makoviychuk, Oliver Kroemer, Dieter Fox.
International Conference on Robotics and Automation (ICRA).
May 2020.
[arXiv]
[Website]
DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System
Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff, Dieter Fox.
International Conference on Robotics and Automation (ICRA).
May 2020.
[arXiv]
[Website]
Homography-Based Deep Visual Servoing Methods for Planar Grasps
Austin S. Wang, Wuming Zhang, Daniel Troniak, Jacky Liang, Oliver Kroemer.
International Conference on Intelligent Robots and Systems (IROS).
November 2019.
[Paper]
Towards Precise Robotic Grasping by Probabilistic Post-grasp Displacement Estimation
Jialiang Zhao, Jacky Liang, Oliver Kroemer.
Field and Service Robotics (FSR).
August 2019.
[arXiv]
GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning
Jacky Liang*, Viktor Makoviychuk*, Ankur Handa*, Nuttapong Chentanez, Miles Macklin, Dieter Fox.
Conference on Robot Learning (CoRL).
October 2018.
[arXiv]
[Website]
Using dVRK Teleoperation to Facilitate Deep Learning of Automation Tasks for an Industrial Robot
Jacky Liang, Jeffrey Mahler, Michael Laskey, Pusong Li, Ken Goldberg.
Conference on Automation Science and Engineering (CASE).
August 2017.
Finalist, Best Student Paper Award.
[PDF]
[Website]
Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics
Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, Richard Doan, Xinyu Liu, Juan Aparicio Ojea, Ken Goldberg. Robotics: Science and Systems (RSS).
Cambridge, MA. July 2017.
[PDF]
[Website]
Design of Parallel-Jaw Gripper Tip Surfaces for Robust Grasping
Menglong Guo, David V. Gealy, Jacky Liang, Jeffrey Mahler, Aimee Goncalves, Stephen McKinley, Ken Goldberg.
International Conference on Robotics and Automation (ICRA).
Singapore, May 2017.
[PDF]
Undergraduate-Led Survey Class to Improve CS Education for New Students
Nathan Zhang*, Jacky Liang*, Amanda Tomlinson*, Frank Boensch, Anant Sahai. SIGCSE ‘20: Proceedings of the 51st ACM Technical Symposium on Computer Science Education. February 2020. [Paper]
Technical Reports
A Modular Robotic Arm Control Stack for Research: Franka-Interface and FrankaPy
Kevin Zhang*, Mohit Sharma*, Jacky Liang*, Oliver Kroemer.
November 2020.
[arXiv]
* Equal Contribution.
Teaching
16-350 Planning Techniques for Robotics
Spring 2021 Teaching Assistant
16-662 Robot Autonomy
Spring 2020 Teaching Assistant
CS188 Introduction to Artificial Intelligence
Spring 2017 Discussion Teaching Assistant
EE16A Design of Information Devices and Systems
Fall 2016, 2017 Discussion Teaching Assistant
Education
Carnegie Mellon University. Robotics Institute. PhD
University of California Berkeley. Electrical Engineering and Computer Science. Bachelor of Science, Class of 2018
Open-Source Libraries
- Language Model Programs - Interactive colab notebook that implements language model programs for a simulated table-top manipulation domain.
- FrankaPy - Python library for controlling the Franka Emika Panda robot arm.
- IsaacGym-Utils - Python library that provides simpler APIs for the Nvidia Isaac Gym robot simulator
- Simple ZMQ - Python library that uses zmq to send arbtirary objects over a network
- Async Savers - Python library for asynchronously saving data in shards
- Data Learning Boilerplate - Boilerplate Python code for collecting data and training neural networks using W&B and PyTorch Lightning
Useful Tools
Assorted Links
- A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
- How to avoid machine learning pitfalls: a guide for academic researchers
- Monte Carlo Gradient Estimation in Machine Learning
- A Recipe for Training Neural Networks
- Policy Search in Continuous Action Domains: an Overview
- A Visual Guide to Evolution Strategies
- Lie Groups for 2D and 3D Transformations
- Optimization inequalities cheatsheet
- Common Academic Experiences No One Talks About: Repeated Rejection, Impostor Syndrome, and Burnout