BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects

This is an implementation of our paper published in CVPR 2023

[Arxiv] [Project page] [Supplemental video]

Abstract

We present a near real-time method for 6-DoF tracking of an unknown object from a monocular RGBD video sequence, while simultaneously performing neural 3D reconstruction of the object. Our method works for arbitrary rigid objects, even when visual texture is largely absent. The object is assumed to be segmented in the first frame only. No additional information is required, and no assumption is made about the interaction agent. Key to our method is a Neural Object Field that is learned concurrently with a pose graph optimization process in order to robustly accumulate information into a consistent 3D representation capturing both geometry and appearance. A dynamic pool of posed memory frames is automatically maintained to facilitate communication between these threads. Our approach handles challenging sequences with large pose changes, partial and full occlusion, untextured surfaces, and specular highlights. We show results on HO3D, YCBInEOAT, and BEHAVE datasets, demonstrating that our method significantly outperforms existing approaches.

Bibtex

@InProceedings{bundlesdfwen2023,
author        = {Bowen Wen and Jonathan Tremblay and Valts Blukis and Stephen Tyree and Thomas M\"{u}ller and Alex Evans and Dieter Fox and Jan Kautz and Stan Birchfield},
title         = {{BundleSDF}: {N}eural 6-{DoF} Tracking and {3D} Reconstruction of Unknown Objects},
booktitle     = {CVPR},
year          = {2023},
}

Data download

• Download pretrained weights of segmentation network, and put it under ./BundleTrack/XMem/saves/XMem-s012.pth

• Download pretrained weights of LoFTR outdoor_ds.ckpt, and put it under ./BundleTrack/LoFTR/weights/outdoor_ds.ckpt

• Download HO3D data. We provide the augmented data that you can download here. Then download YCB-Video object models from here. Finally, make sure the structure is like below, and update your root path of HO3D_ROOT at the top of BundleTrack/scripts/data_reader.py

  HO3D_v3
    ├── evaluation
    ├── models
    └── masks_XMem
  

Docker/Environment setup

• Build the docker image (this only needs to do once and can take some time).

cd docker
docker build --network host -t nvcr.io/nvidian/bundlesdf .

• Start a docker container the first time

cd docker && bash run_container.sh

# Inside docker container, compile the packages which are machine dependent
bash build.sh

Run on your custom data

• Prepare your RGBD video folder as below (also refer to the example milk data). You can find an example milk data here for testing.

root
  ├──rgb/    (PNG files)
  ├──depth/  (PNG files, stored in mm, uint16 format. Filename same as rgb)
  ├──masks/       (PNG files. Filename same as rgb. 0 is background. Else is foreground)
  └──cam_K.txt   (3x3 intrinsic matrix, use space and enter to delimit)

Due to license issues, we are not able to include XMem in this codebase for running segmentation online. If you are interested in doing so, please download the code separately and add a wrapper in segmentation_utils.py.

• Run your RGBD video (specify the video_dir and your desired output path). There are 3 steps.

# 1) Run joint tracking and reconstruction
python run_custom.py --mode run_video --video_dir /home/bowen/debug/2022-11-18-15-10-24_milk -