Carview!

CARVIEW

MOTORHOMES

Select Language

HTTP/2 301 server: GitHub.com content-type: text/html location: https://yixchen.github.io/YouRefIt/ x-github-request-id: F668:21D6A4:859678:9602A2:6952056F accept-ranges: bytes age: 0 date: Mon, 29 Dec 2025 04:37:04 GMT via: 1.1 varnish x-served-by: cache-bom-vanm7210033-BOM x-cache: MISS x-cache-hits: 0 x-timer: S1766983024.436919,VS0,VE201 vary: Accept-Encoding x-fastly-request-id: 03496a871a5a8512d6a7c780325ad04c5c0d1378 content-length: 162 HTTP/2 200 server: GitHub.com content-type: text/html; charset=utf-8 last-modified: Tue, 09 Jul 2024 05:09:28 GMT access-control-allow-origin: * strict-transport-security: max-age=31556952 etag: W/"668cc608-260e" expires: Mon, 29 Dec 2025 04:47:04 GMT cache-control: max-age=600 content-encoding: gzip x-proxy-cache: MISS x-github-request-id: 106A:36A0B4:846365:94D0D6:69520570 accept-ranges: bytes age: 0 date: Mon, 29 Dec 2025 04:37:04 GMT via: 1.1 varnish x-served-by: cache-bom-vanm7210033-BOM x-cache: MISS x-cache-hits: 0 x-timer: S1766983025.657642,VS0,VE223 vary: Accept-Encoding x-fastly-request-id: d0af612988c68820dab311f635718edeafad5903 content-length: 3133 YouRefIt

Abstract

We study the machine's understanding of embodied reference: One agent uses both language and gesture to refer to an object to another agent in a shared physical environment. Of note, this new visual task requires understanding multimodal information with perspective-taking to identify which object is being referred to. To tackle this problem, we introduce YouRefIt, a new crowd-sourced, real-world dataset of embodied reference; the dataset contains 4,195 unique reference clips in 432 indoor scenes. To the best of our knowledge, this is the first embodied reference dataset that affords us to study referring expressions in real-world scenes for understanding referential behavior, human communications, and human-robot interaction. We further devise two benchmarks for image-based and video-based embodied reference understanding. Comprehensive base-lines and extensive experiments provide the very first result of machine perception on how the referring expressions and gestures affect the embodied reference understanding. Our results provide essential evidence that gestural information is as critical as language information in understanding the embodied reference.

Paper

YouRefIt: Embodied Reference Understanding with Language and Gesture
Yixin Chen, Qing Li, Deqian Kong, Yik Lun Kei, Tao Gao, Yixin Zhu, Song-Chun Zhu, Siyuan Huang
The IEEE International Conference on Computer Vision (ICCV) 2021 (oral)
Paper / Supplementary / Code / Dataset / Bibtex

Team

Yixin Chen

Qing Li

Deqian Kong

Yik Lun Kei

Song-Chun Zhu

Tao Gao

Yixin Zhu

Siyuan Huang

Bibtex

@inProceedings{chen2021yourefit,
title={YouRefIt: Embodied Reference Understanding with Language and Gesture},
author = {Chen, Yixin and Li, Qing and Kong, Deqian and Kei, Yik Lun and Zhu, Song-Chun and Gao, Tao and Zhu, Yixin and Huang, Siyuan},
booktitle={The IEEE International Conference on Computer Vision (ICCV),
year={2021}
}

Original Source | Taken Source