HTTP/2 301
server: GitHub.com
content-type: text/html
location: https://msieb.com/visual-entity-graphs
x-github-request-id: 7ED2:2C10E1:92F2BF:A4C826:6952C069
accept-ranges: bytes
age: 0
date: Mon, 29 Dec 2025 17:54:51 GMT
via: 1.1 varnish
x-served-by: cache-bom-vanm7210068-BOM
x-cache: MISS
x-cache-hits: 0
x-timer: S1767030891.898958,VS0,VE200
vary: Accept-Encoding
x-fastly-request-id: 597d8455fc339e0e5b638af7d66c183edd494158
content-length: 162
HTTP/2 301
server: GitHub.com
content-type: text/html
location: https://msieb.com/visual-entity-graphs/
x-github-request-id: 080A:3946E9:91AB74:A37EE9:6952C06B
accept-ranges: bytes
age: 0
date: Mon, 29 Dec 2025 17:54:51 GMT
via: 1.1 varnish
x-served-by: cache-bom-vanm7210038-BOM
x-cache: MISS
x-cache-hits: 0
x-timer: S1767030891.188619,VS0,VE198
vary: Accept-Encoding
x-fastly-request-id: 7b5fb8ef344c9f5a1d2ff38fb47cd9cd0090f2cf
content-length: 162
HTTP/2 200
server: GitHub.com
content-type: text/html; charset=utf-8
last-modified: Mon, 15 Jul 2019 16:24:59 GMT
access-control-allow-origin: *
strict-transport-security: max-age=31556952
etag: W/"5d2ca8db-36bd"
expires: Mon, 29 Dec 2025 18:04:51 GMT
cache-control: max-age=600
content-encoding: gzip
x-proxy-cache: MISS
x-github-request-id: 110D:2B0FD4:92E999:A4BC7A:6952C06A
accept-ranges: bytes
age: 0
date: Mon, 29 Dec 2025 17:54:51 GMT
via: 1.1 varnish
x-served-by: cache-bom-vanm7210038-BOM
x-cache: MISS
x-cache-hits: 0
x-timer: S1767030891.400644,VS0,VE233
vary: Accept-Encoding
x-fastly-request-id: 8b63c879e989828800d42702e85e10ab0fe757da
content-length: 3505
Graph-Structured Visual Imitation
Graph-Structed Visual Imitation
|
|
Visual entity graph (VEG) representation of human demonstration and robot imitation for two timesteps of a typical imitation. Corresponding nodes in the human demonstration (top) and robot imitation (bottom) share the same color. Our graphs are hierarchical. Edges exist between object and robot/human hand nodes and point feature nodes, and are added/deleted dynamically over time based on the motion saliency (as shown in the figure with solid/dashed lines).
|
Abstract
We cast visual imitation as a visual correspondence problem. Our robotic agent is rewarded when its actions result in better matching of relative spatial configurations for corresponding visual entities detected in its workspace and teacher's demonstration. We build upon recent advances in Computer Vision,such as human finger keypoint detectors, object detectors trained on-the-fly with synthetic augmentations, and point detectors supervised by viewpoint changes and learn multiple visual entity detectors for each demonstration without human annotations or robot interactions. We empirically show the proposed factorized visual representations of entities and their spatial arrangements drive successful imitation of a variety of manipulation skills within minutes, using a single demonstration and without any environment instrumentation. It is robust to background clutter and can effectively generalize across environment variations between demonstrator and imitator, greatly outperforming unstructured non-factorized full-frame CNN encodings of previous works.
|
|
We use human hand keypoint detectors, multi-viewpoint feature learning, and synthetic image generation for on-the-fly object detector training from only a few object mask examples. Using a manually designed mapping between the human hand and the robot, the visual
detectors can effectively bridge the visual gap between demonstrator and imitator environment, are robust to background clutter, and generalize across different object instances.
|
|
|
Example of the pouring imitation task. Top row shows demonstration, second row shows imitation, and third row shows generalization to a novel object instance.
|
Supplementary Videos
Pushing Task: we train the robot to push the yellow hexagon to purple ring along different trajectories, and show generalization to pushing a different object
Stacking Task: we train the robot to place the yellow hexagon on the purple ring, and show generalization to a different object and different background
Pouring Task: we train the robot to pour liquid from the orange can to mug, and show generalization of this task to a different can using pixel features
Below is a video of all demonstrated tasks and their successful imitation.
