HTTP/2 301 server: GitHub.com content-type: text/html location: https://ai4ce.github.io/SeeDo/ x-github-request-id: 7E24:36A0B4:82ED9F:932939:6951EB9B accept-ranges: bytes age: 0 date: Mon, 29 Dec 2025 02:46:51 GMT via: 1.1 varnish x-served-by: cache-bom-vanm7210067-BOM x-cache: MISS x-cache-hits: 0 x-timer: S1766976412.763491,VS0,VE198 vary: Accept-Encoding x-fastly-request-id: 1c849c53af4437b14791a65a6f2ae0f6ab1981de content-length: 162 HTTP/2 200 server: GitHub.com content-type: text/html; charset=utf-8 last-modified: Tue, 15 Oct 2024 05:05:42 GMT access-control-allow-origin: * strict-transport-security: max-age=31556952 etag: W/"670df826-5918" expires: Mon, 29 Dec 2025 02:56:52 GMT cache-control: max-age=600 content-encoding: gzip x-proxy-cache: MISS x-github-request-id: D49A:2D64E0:81ED49:9226E1:6951EB9B accept-ranges: bytes age: 0 date: Mon, 29 Dec 2025 02:46:52 GMT via: 1.1 varnish x-served-by: cache-bom-vanm7210067-BOM x-cache: MISS x-cache-hits: 0 x-timer: S1766976412.975165,VS0,VE217 vary: Accept-Encoding x-fastly-request-id: 2faa725c38a58be9e3f33bb4bd5869aeeafecd96 content-length: 5375

VLM See, Robot Do:

Human Demo Video to Robot Action Plan via Vision Language Model

Beichen Wang^*1, Juexiao Zhang^*1, Shuwen Dong^†1 Irving Fang^†1, Chen Feng¹,

¹ New York University,
^* Equal contribution, first authors. ^† Equal contribution, second authors.

arXiv Code

TLDR

Interpret human demonstration videos and generate robot action plans using a pipeline of keyframe selection, visual perception and vision language model reasoning.

---

---

Method

1. Keyframe Selection 2. Visual Perception 3. VLM Reasoning

Module 1: Keyframe Selection
We use APIs from the MediaPipe to detect the hand keypoints and calculate the speed of the hand.
The speed plot is then interpolated to be continous and the valleys are used as the keyframe selections.

---

Data and demos

We collected a dataset of human demonstration videos in three diverse catogories: vegetable organization, garment organization, and wooden block stacking.

Below are the data and corresponding results.

Vegetable Organization Garment Organization Wooden Block Stacking

Human demonstration
Here is a demonstration video for vegetable organization. The videos illustrate how a human arrange the vegetable toys into specific containers one by one.

Robot execution
In this video, the robot executes the vegetable organization task in the same order as the human demonstrates.

Coming Soon