MobileBrick: Building LEGO for 3D Reconstruction on Mobile Devices

Kejie Li1 Jia-Wang Bian1, Robert Castle2, Philip H.S. Torr1, Victor Adrian Prisacariu1
1 University of Oxford 2 Apple

Arxiv Data Code

Abstract

Generate high-quality 3D ground-truth shapes for reconstruction evaluation is extremely challenging because even 3D scanners can only generate pseudo ground-truth shapes with artefacts. We propose a novel data capturing and 3D annotation pipeline to obtain precise 3D ground-truth shapes without relying on expensive 3D scanners. The key to creating the precise 3D ground-truth shapes is using LEGO models, which are made of LEGO bricks with known geometry. The MobileBrick dataset provides a unique opportunity for future research on high-quality 3D reconstruction thanks to two distinctive features: 1) A large number of RGBD sequences with precise 3D ground-truth annotations. 2) The RGBD images were captured using mobile devices so algorithms can be tested in a realistic setup for mobile AR applications.

3D Object Alignment

We showcase the quality of GT shape alignment to the image sequence by projecting the 3D model to the RGB images (shown as "3D Model Projection"). "GT depth" is the depth maps rendered from 3D model at the same viewpoint. "RGB Image" and "ARKit Depth" are the high-res RGB images and low-res depth maps provided by ARKit.

Benchmark Results

Methods Input Data σ =2.5mm σ =5mm Chamfer (mm)
Accu. (%) Rec. (%) F1 Accu. (%) Rec. (%) F1
TSDF-Fusion Depth 42.07 22.21 28.77 73.46 42.75 53.29 13.78
BNV-Fusion Depth 41.77 25.96 33.27 71.20 47.09 55.11 9.60
Neural-RGBD RGBD 20.61 10.66 13.67 39.62 22.06 27.66 22.78
COLMAP RGB 74.89 68.20 71.08 93.79 84.53 88.71 5.26
Vis-MVSNet RGB 79.83 47.25 58.32 97.35 65.90 77.49 9.27
Vis-MVSNet (finetuned) RGB 75.64 53.64 62.01 96.03 72.42 81.89 9.52
NeRF RGB 47.11 40.86 43.55 78.07 69.03 73.45 7.98
NeuS RGB 77.35 70.85 73.74 93.33 86.11 89.30 4.74
Voxurf RGB 78.44 72.41 75.13 93.95 87.53 90.41 4.71
The methods achiving the best and second best performance are denoted in Green and Yellow respectively

BibTeX

    @article{li2023mobilebrick,
      author = {Kejie Li, Jia-Wang Bian, Robert Castle, Philip H.S. Torr, Victor Adrian Prisacariu},
      title = {MobileBrick: Building LEGO for 3D Reconstruction on Mobile Devices},
      journal={arXiv preprint arXiv:2303.01932},
      year={2023}
    }