XVO: Generalized Visual Odometry via Cross-Modal Self-Training

Our approach is inspired by how humans learn general representations through observation of large amouts of multi-modal data, and the successes witnessed in computer vision through multi-task frameworks and auxiliary learning.
Humans can flexibly estimate motion in arbitrary conditions through a general understanding of salient scene properties (e.g., object sizes). This general understanding is developed over large amounts of perceptual data, often multi-modal in nature in arbitrary conditions through a general understanding of salient scene properties. For instance, cross-modal information processing between audio and video has been shown to play a role in spatial reasoning and proprioception. Drawing from this human learning paradigm, our approach leverages extensive multimodal data through a semi-supervised training process, incorporating various auxiliary prediction tasks, including segmentation, flow, depth, and audio, to achieve generalized visual odometry without any camera intrinsic or extrinsic information in diverse environment under arbitrary conditions. Our approach is also motivated by the successes witnessed in computer vision through multi-task frameworks, where auxiliary tasks, in addition to the primary learning objective, serve as a regulation to reduce the overfitting and enrich the feature representations.

The figure above illustrates the importance of audio in estimating vehicle acceleration and braking. The frame is consistent with the red arrow marked on the waveform. On the left, the audio amplitude decreases and maintains a low level when the vehicle is going to wait for traffic lights. On the right, the audio experiences many ups and downs associated with acceleration and braking in a narrow urban area.

Our proposed framework comprises the following steps: (1) uncertainty-aware training of an initial (i.e., teacher) VO model; (2) pseudo-labeling with the removal of low-confidence and potentially noisy samples; (3) self-training with pseudo-labeled and auxiliary prediction tasks of a robust VO student model. Specifically, we first train an ego-motion prediction teacher model over a small initial dataset, e.g., nuScenes. We then expand the original dataset through pseudo-labeling of in-the-wild videos. After that, we employ multiple auxiliary prediction tasks, including segmentation, flow, depth, and audio, as part of the semi-supervised training process Finally, we leverage uncertainty-based filtering of potentially noisy pseudo labels and train a robust student model.