HTTP/2 200 server: GitHub.com content-type: text/html; charset=utf-8 last-modified: Fri, 17 Mar 2023 06:43:37 GMT access-control-allow-origin: * strict-transport-security: max-age=31556952 etag: W/"64140c19-a960" expires: Mon, 29 Dec 2025 19:55:30 GMT cache-control: max-age=600 content-encoding: gzip x-proxy-cache: MISS x-github-request-id: F474:3A7A40:93A286:A5A97A:6952DA59 accept-ranges: bytes age: 0 date: Mon, 29 Dec 2025 19:45:30 GMT via: 1.1 varnish x-served-by: cache-bom-vanm7210064-BOM x-cache: MISS x-cache-hits: 0 x-timer: S1767037531.542874,VS0,VE222 vary: Accept-Encoding x-fastly-request-id: b7bcab9d17cb3074fd9f6581690e4037b0ca5a92 content-length: 5295

RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control

Zhenggang Tang¹, Balakumar Sundaralingam², Jonathan Tremblay², Bowen Wen², Ye Yuan¹, Stephen Tyree², Charles Loop², Alexander Schwing¹, Stan Birchfield²,

¹University of Illinois Urbana Champaign & ²NVIDIA

Paper Video Data

Abstract

We present a system for collision-free control of a robot manipulator that uses only RGB views of the world. Perceptual input of a tabletop scene is provided by multiple images of an RGB camera (without depth) that is either handheld or mounted on the robot end effector. A NeRF-like process is used to reconstruct the 3D geometry of the scene, from which the Euclidean full signed distance function (ESDF) is computed. A model predictive control algorithm is then used to control the manipulator to reach a desired pose while avoiding obstacles in the ESDF. We show results on a real dataset collected and annotated in our lab.

Video

Results for Scene 1

gt our colmap isdf voxblox

gt our colmap isdf voxblox

Results for Scene 2

gt our colmap isdf voxblox

gt our colmap isdf voxblox

Results for Scene 3

gt our colmap isdf voxblox

gt our colmap isdf voxblox

BibTeX

@inproceedings{tang2023icra:rgbonly,
  author = "Zhenggang Tang and Balakumar Sundaralingam and Jonathan Tremblay and Bowen Wen and Ye Yuan and Stephen Tyree and Charles Loop and Alexander Schwing and Stan Birchfield",
  title = "{RGB}-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control",
  booktitle = "ICRA",
  year = 2023
}