This isn't meant to be something that works in 5 years--it's meant to be something that works now. Below, we show that the pipeline can create datasets for several different computer vision tasks--each capable of training existing approaches to competitive or better than state-of-the-art performance. The results above look comparable to the original, and the quanitative results tell the same story: the version trained on the starter data outperformed the original mix of 10 existing depth-specific datasets when evaluated zero-shot on NYU (Silberman et al. 2014) and OASIS (Chen et al. 2020). For surface normal estimation (below), the generated dataset also yield networks that show state-of-the-art zero-shot performance on OASIS. Along one of the metrics, the network gets human-level performance (though not for other metrics). The networks don't seem human-level, but they are qualitatively better than fancier approaches trained on existing datasets: The annotator can be used for many downstream tasks: it produces labels for 21 different mid-level cues. These vision-based cues can also be used to improve perfomance for visuomotor tasks: for example navigation ones, manipulation ones, and it works with actual physical robots.

For a complete list of labels and how they are produced, see the annotator GitHub. To try out the pretrained models, upload an image to a live demo or download the PyTorch weights. Or train your own model using the starter data and dataloaders.


Most existing vision datasets capture images once. Someone or something decides which images to take and which not to, what sensors to use, what subjects to frame, etc. After capture, these design decisions become fixed and difficult to change. These biases in the training data become encoded in models trained on the data, and affect the resulting models' ability to generalize to other types of situations.
By capturing as much information as possible and then parametrically resampling that data into 3D images, we can probe the effects of different sampling distributions and data domains. For example, previous research has identified various types of selection bias such as photographer’s bias, viewpoint bias. Choice of sensor (e.g. RGB vs. LIDAR) also affects what information is available to the model.
These choices have real impact. For example, selecting different aperture sizes changes the makeup of images (below and left). In effect, making an dataset more or less object-centric. Play with some of these effects in our dataset design demo or make one yourself with one of the one-line examples in our annotator quickstart.


Models pretrained on ImageNet are considered the default starting point for almost any computer vision project. But what causes a model pretrained on ImageNet to transfer better to other downstream tasks compared to, say, a depth model trained on NYU? Is it that classification is better than depth estimation?
ImageNet features are not always best (in vision, in robotics), but determining why an ImageNet classification pretraining is often better than NYU depth estimation pretraining is difficult because ImageNet has no depth labels. So the effect of the pretraining data cannot be separated from the pretraining task.
With the Omnidata pipeline, we can annotate the same dataset for both tasks, in order to determine whether it is the classification pretraining or the image distribution that is doing the heavy lifting. Similarly, the pipeline gives us full control over dataset generation in order to determine the impact of single- vs multi-view training, using correspondences, geometry, camera pose, etc.