Depth upsampling

Data download

To download the data please follow the data documentation

Data organization and format of input data

The dataset includes 4 types of assets and metadata:

color - the RGB images (1920x1440)
highres_depth - the ground-truth depth image projected from the mesh generated by Faro’s laser scanners (1920x1440)
lowres_depth - the depth image acquired by AppleDepth Lidar (256x192)
confidence - the confidence of the AppleDepth depth image (256x192)
metadata.csv - meta data per video (i.e. sky direction - (up/down/left/right))
val_attributes.csv - attributes per sample (i.e. transparent_or_reflective - if True, the image includes a transparent or reflective objects). Manually annotated and only relevant for the Validation bin.

Data documentation describe the format of each one of the asset.

ARKitScenes/depth_upsampling/
├── Training                                # training bin assets folder  
│   ├── 41069021                            # video_id assets folder
│   │   ├── color                           # color assets folder
│   │   │   ├── 41069021_305.244.png        # color frames
│   │   │   ├── 41069021_307.343.png
│   │   │   ├── 41069021_309.742.png
│   │   │   └── ...
│   │   ├── highres_depth                   # highres_depth folder
│   │   │   ├── 41069021_305.244.png        # highres_depth frames
│   │   │   ├── 41069021_307.343.png
│   │   │   ├── 41069021_309.742.png
│   │   │   └── ...
│   │   ├── lowres_depth                    # lowres_depth folder
│   │   │   ├── 41069021_305.244.png        # lowres_depth frames
│   │   │   ├── 41069021_307.343.png
│   │   │   ├── 41069021_309.742.png
│   │   │   └── ...
│   │   └── confidence                      # confidence folder
│   │       ├── 41069021_305.244.png        # confidence frames
│   │       ├── 41069021_307.343.png
│   │       ├── 41069021_309.742.png
│   │       └── ...
│   ├──
│   └── ...
└── Validation                              # validation bin assets folder
    └── ...

Creating a python environment

The packages required for training depth upsampling are listed in the file requirements.txt, to install them run

cd depth_upsampling
pip install -r requirements.txt

Visualizing depth upsampling assets

To view upsampling assets you can use the following script: (note that first you need to [download](#Data download) the dataset)

python3 depth_upsampling/sample_vis.py YOUR_DATA_DIR/ARKitScenes --split [train/val] --sample_id SAMPLE_ID

for example to visualize a sample from validation bin you can run:

python3 depth_upsampling/sample_vis.py YOUR_DATA_DIR/ARKitScenes --split val --sample_id 41069021_305.244.png

training depth upsampling

You can train the upsampling networks by running

python train.py --network [MSG/MSPF] --upsample_factor [2/4/8]

The training script will print to the screen the metrics once every 5k iterations. To view the results on tensorboard you can add a tensorboard port parameter --tbp some_port_number to the train.py input parameters. This will automatically open a tensorboard process on a subprocess.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!