PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Tracking
TL;DR A particle filter tracks an object's 6D pose in video by sampling translations and keeping a full rotation distribution for each particle, scored by matching auto-encoder embeddings against a precomputed codebook. It captures symmetry ambiguity without symmetry labels and doubles RGB recall on T-LESS over single-frame estimation.
What they do
Single-image 6D pose estimators return one pose with no uncertainty, which is not enough for symmetric or occluded objects. PoseRBPF tracks the full pose posterior over a video with a Rao-Blackwellized particle filter: each particle samples a 3D translation and carries a discrete distribution over 3D rotation. Objects with any kind of symmetry are handled without labeling the symmetry by hand.
How it works
Factorization. The posterior splits into P(T Z) and P(R T,Z). Rotation is discretized at 5 degrees in azimuth, elevation and in-plane angle, 191,808 bins per particle. - Codebook. A per-object auto-encoder, trained on renders over random COCO backgrounds with occluders, maps a crop to a clean synthetic rendering. Embeddings of renders at every discretized rotation, at a canonical distance, form a precomputed codebook.
- Update. Each particle’s translation sets a square RoI scaled by depth. Cosine similarity between its embedding and the codebook gives the rotation likelihood; the sum gives the particle weight, since off-center or wrong-scale crops match poorly. Motion: constant velocity for translation, Gaussian convolution over the rotation grid.
- RGB-D. Optionally, the object rendered at each particle’s best rotation is compared with the depth image for a visibility and depth-discrepancy score.
Results
- Speed: 20.3 fps with 50 particles on RGB, 6.1 fps with 200 (5.0 with RGB-D).
- YCB Video: initialized by PoseCNN (1.03 re-initializations per sequence on average), it improves accuracy, especially on symmetric objects like the bowl and foam brick; depth helps further.
- T-LESS (100 particles): tracking doubles RGB recall over the single-frame auto-encoder method it builds on. Depth raises recall by about 76% and beats that method refined with ICP by 28%.
- Uncertainty: for scissors, the ground truth is within 20 degrees of a rotation in the top 20% of the distribution 80% of the time, against about 60% for a Gaussian around PoseCNN.
Limitations
Each object needs a 3D model and its own trained auto-encoder and codebook, and accuracy drops when the model differs from the real object (the wood block). A detector is needed to initialize and re-initialize after heavy occlusion. Frame rate falls to 3.1 fps at 400 particles.
Videos
Figures
Abstract
Tracking 6D poses of objects from videos provides rich information to a robot in performing different tasks such as manipulation and navigation. In this work, we formulate the 6D object pose tracking problem in the Rao-Blackwellized particle filtering framework, where the 3D rotation and the 3D translation of an object are decoupled. This factorization allows our approach, called PoseRBPF, to efficiently estimate the 3D translation of an object along with the full distribution over the 3D rotation. This is achieved by discretizing the rotation space in a fine-grained manner, and training an auto-encoder network to construct a codebook of feature embeddings for the discretized rotations. As a result, PoseRBPF can track objects with arbitrary symmetries while still maintaining adequate posterior distributions. Our approach achieves state-of-the-art results on two 6D pose estimation benchmarks. A video showing the experiments can be found at https://youtu.be/lE5gjzRKWuA
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.