| CARVIEW |
Dobb·E
An open-source, general framework for learning household robotic manipulation
109 tasks
10 NYC homes
81% success rate
20 minutes to learn a new task
Throughout history, we have successfully integrated various machines into our homes. Dishwashers, laundry machines, stand mixers, and robot vacuums are just a few recent examples. However, these machines excel at performing only a single task effectively. The concept of a “generalist machine” in homes - a domestic assistant that can adapt and learn from our needs, all while remaining cost-effective - has long been a goal in robotics that has been steadily pursued for decades. In this work, we initiate a large-scale effort towards this goal by introducing Dobb·E, an affordable yet versatile general-purpose system for learning robotic manipulation within household settings. Dobb·E can learn a new task with only five minutes of a user showing it how to do it, thanks to a demonstration collection tool ("The Stick") we built out of cheap parts and iPhones. We use the Stick to collect 13 hours of data in 22 homes of New York City, and train Home Pretrained Representations (HPR). Then, in a novel home environment, with five minutes of demonstrations and fifteen minutes of adapting the HPR model, we show that Dobb·E can reliably solve the task on the Stretch, a mobile robot readily available on the market. Across roughly 30 days of experimentation in homes of New York City and surrounding areas, we test our system in 10 homes, with a total of 109 tasks in different environments, and finally achieve a success rate of 81%. Beyond success percentages, our experiments reveal a plethora of unique challenges absent or ignored in lab robotics. These range from effects of strong shadows to variable demonstration quality by non-expert users. With the hope of accelerating research on home robots, and eventually seeing robot butlers in every home, we open-source Dobb·E software stack and models, our data, and our hardware designs.
Videos
Dobb·E in action
In 10 homes of New York City, Dobb·E attempted 109 tasks. Here are sample rollouts from each of the tasks.
Hardware
The Stick
We believe one of the largest roadblocks to safe and scalable progress in home robotics, especially in imitation learning based approaches, is the lack of a cheap, ergonomic, and easy way to collect demonstrations for robots.
To address this, we built the Stick, a demonstration collection tool we built out of a $25 Reacher-grabber stick, some 3D printed parts, and an iPhone.
Dataset
Homes of New York (HoNY)
Homes of New York (HoNY) is a dataset containing 13 hours of interactions at 22 different homes of New York City collected with the Stick. The dataset contains RGB and depth videos at 30 fps, as well as full action annotations for 6D pose of the gripper as well as the gripper's opening angle normalized between (0, 1).
Model
Home Pretrained Representations (HPR)
Home Pretrained Representation (HPR) is a model pre-trained on the HoNY dataset that we used to initialize a robot policy to perform a new task in a novel enviroment. HPR is a ResNet-34 model trained on the HoNY dataset using the MoCo-v3 self-supervised learning objective.
During deployment,we used HPR to initialize a policy, the trunk of which was simply our pretrained ResNet-34 model followed by two linear layers on top.
🤗 Get the model at HuggingfaceOr if you are using 🤗 Pytorch Image Models (TIMM), you can simply start using it in a couple of lines:
import timm
model = timm.create_model(
"hf-hub:notmahi/dobb-e",
pretrained=True
)
import timm
model = timm.create_model("hf-hub:notmahi/dobb-e", pretrained=True)
Paper
On Bringing Robots Home
@article{shafiullah2023bringing,
title={On bringing robots home},
author={Shafiullah, Nur Muhammad Mahi and Rai, Anant and Etukuru, Haritheja and Liu, Yiqian and Misra, Ishan and Chintala, Soumith and Pinto, Lerrel},
journal={arXiv preprint arXiv:2311.16098},
year={2023}
}