BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects

Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas Müller, Alex Evans, Dieter Fox, Jan Kautz, Stan Birchfield

Seminal March 2023

Tracking and reconstruction with a human body, a robot arm and a human hand moving the object.

TL;DR BundleSDF tracks the 6-DoF pose of an unknown rigid object in an RGBD video and reconstructs its textured mesh at the same time, given only a mask in the first frame. On the HO3D hand-object benchmark it reaches 92.6% ADD versus 66.0% for the next best method, with 0.57 cm chamfer error.

What they do

BundleSDF tracks an unknown rigid object through a monocular RGBD video and reconstructs its shape and texture at the same time. The only input besides the video is a 2D object mask in the first frame: no CAD model, no category prior, and no model of whatever moves the object (hand, body or robot arm). Processing is causal, and the tracker runs at about 10 Hz.

How it works

Coarse pose. A video segmentation network masks the object in each frame. Transformer-based feature matching plus RANSAC on depth gives a pose relative to the previous frame.
Memory pool and online pose graph. Keyframes enter the pool only if they add a new viewpoint (rotation distance ignoring in-plane rotation). Each new frame is optimized jointly with up to 10 memory frames using correspondence, point-to-plane and SDF losses, solved by Gauss-Newton in CUDA.
Neural Object Field. In a parallel thread, a hash-encoded neural SDF with an appearance network trains on all memory frames while refining their poses (about 6.7 s per round). The refined poses and the SDF go back to the tracker to limit drift.
Hybrid SDF. Rays from background-mask pixels or missing depth are treated as uncertain free space with a small positive SDF target, so noisy masks and hand occlusions do not bias the shape.

Results

HO3D (hand-held objects): 92.62% ADD and 0.57 cm chamfer distance, against 66.01% and 52.05 cm for BundleTrack and 33.36% and 30.84 cm for DROID-SLAM.
YCBInEOAT (robot-arm manipulation): best ADD-S (93.77%) and chamfer distance (1.16 cm); ADD of 86.95% is on par with BundleTrack’s 87.26%.
BEHAVE (full-body interaction, single far-range view): 67.52% ADD versus 45.03% for BundleTrack.
Ablations on HO3D: without the memory, ADD falls to 56.96%; without the Neural Object Field, to 76.69%.

Limitations

Tracking fails when several challenges co-occur (severe occlusion, segmentation errors, little texture or geometry), and symmetric shapes can give a wrong pose when the object reappears. Depth is required, which rules out objects like transparent ones, and the object must be rigid. Parts never seen in the video are not reconstructed.

Videos

A hand-held object tracked in a kitchen while its mesh builds up alongside.

A milk jug turned by hand on the floor, tracked and reconstructed online.

Figures

From Figure 1: Given an RGBD sequence and an object mask in the first frame only, the method tracks the 6-DoF pose of... — From Figure 1: Given an RGBD sequence and an object mask in the first frame only, the method tracks the 6-DoF pose of an unknown object and reconstructs it, across flat and untextured surfaces, specular highlights, thin structures, severe occlusion, and human hand, body or robot arm interaction. The meshes shown are output directly by the method.

From Figure 2: Framework overview. Features matched between consecutive segmented frames give a coarse pose. Some pos... — From Figure 2: Framework overview. Features matched between consecutive segmented frames give a coarse pose. Some posed frames are kept in a memory pool; a pose graph built from a subset of the pool refines them jointly with the current pose. In a separate thread, all memory frames train a Neural Object Field that models geometry and texture while adjusting their poses.

From Figure 4: The three strongest methods on HO3D. Left: pose tracking, with the object contour (cyan) rendered at t... — From Figure 4: The three strongest methods on HO3D. Left: pose tracking, with the object contour (cyan) rendered at the estimated pose; in the 2nd column the prediction corrects an error in the ground truth. Right: front and back views of each final reconstruction; drift in DROID-SLAM and BundleTrack leaves their meshes wrongly rotated.

Abstract

We present a near real-time method for 6-DoF tracking of an unknown object from a monocular RGBD video sequence, while simultaneously performing neural 3D reconstruction of the object. Our method works for arbitrary rigid objects, even when visual texture is largely absent. The object is assumed to be segmented in the first frame only. No additional information is required, and no assumption is made about the interaction agent. Key to our method is a Neural Object Field that is learned concurrently with a pose graph optimization process in order to robustly accumulate information into a consistent 3D representation capturing both geometry and appearance. A dynamic pool of posed memory frames is automatically maintained to facilitate communication between these threads. Our approach handles challenging sequences with large pose changes, partial and full occlusion, untextured surfaces, and specular highlights. We show results on HO3D, YCBInEOAT, and BEHAVE datasets, demonstrating that our method significantly outperforms existing approaches. Project page: https://bundlesdf.github.io

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong