| CARVIEW |
PLA: Language-Driven Open-Vocabulary 3D Scene Understanding
CVPR 2023
*Indicates Equal Contribution, † indicates Equal Corresponding
vending machine
piano
stove
working office
library
Hierarchical novel concepts: "bathroom", "kitchen"
Fine-grained novel concepts: "monitor", "blackboard"
Abstract
Open-vocabulary scene understanding aims to localize and recognize unseen categories beyond the annotated label space. The recent breakthrough of 2D open-vocabulary perception is largely driven by Internet-scale paired image-text data with rich vocabulary concepts. However, this success cannot be directly transferred to 3D scenarios due to the inaccessibility of large-scale 3D-text pairs. To this end, we propose to distill knowledge encoded in pre-trained vision-language (VL) foundation models through captioning multi-view images from 3D, which allows explicitly associating 3D and semantic-rich captions. Further, to facilitate coarse-to-fine visual-semantic representation learning from captions, we design hierarchical 3D-caption pairs, leveraging geometric constraints between 3D scenes and multi-view images. Finally, by employing contrastive learning, the model learns language-aware embeddings that connect 3D and text for open-vocabulary tasks. Our method not only remarkably outperforms baseline methods by 25.8% ~ 44.7% hIoU and 14.5% ~ 50.4% hAP50 on open-vocabulary semantic and instance segmentation, but also shows robust transferability on challenging zero-shot domain transfer tasks.
Approach
Image-bridged Point-language Association
We utilize multi-view images of a 3D scene as a bridge to access knowledge encoded in vision-language models. Text discritions are first generated by a powerful image-captioning model, and is then associated with a set of points in the 3D scene with geometry constraints between images and 3D point clouds. We present hierarchical scene-level, view-level and entity-level point-language association manners to elicit coarse-to-fine semantic-rich language supervision.
Langauge-driven 3D Scene Understanding Framework
Different from the close-set network, the learnable semantic head is replaced by category embeddings encoded by a text encoder from category names. Binary head is to rectify semantic scores with base and novel probability as condition. Instance head is tailored to instance segmentation. Most importantly, to endow the model with rich semantic space to improve open-vocabulary capability, we supervise point embeddings with caption embeddings based on point-language association.
Visualizations
Hierarchical Point-language Association
BibTeX
@inproceedings{ding2022language,
title={PLA: Language-Driven Open-Vocabulary 3D Scene Understanding},
author={Ding, Runyu and Yang, Jihan and Xue, Chuhui and Zhang, Wenqing and Bai, Song and Qi, Xiaojuan},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2023}
}