PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space
TL;DR PointNet++ applies PointNet recursively to nested local neighborhoods of a point cloud, so it learns local features at growing scales, and its multi-scale layers trained with random point dropout cope with uneven sampling density. It reaches 90.7% on ModelNet40 from 1024 xyz points, against 89.2% for PointNet.
What they do
PointNet encodes each point on its own and max-pools everything into one global feature, so it never captures local structure. PointNet++ adds the hierarchy that CNNs get from growing receptive fields. It partitions the points into overlapping local regions by metric distance, encodes each region with a small PointNet, and repeats on the coarser set. Real scans have uneven density, so it also learns how to weight neighborhoods of different sizes.
How it works
- Set abstraction level. Farthest point sampling picks centroids, ball query groups the points within a radius of each, and a mini-PointNet encodes each group in coordinates relative to its centroid.
- Density-adaptive grouping. Multi-scale grouping (MSG) runs PointNets at several radii and concatenates them. Multi-resolution grouping (MRG) concatenates a summary of lower-level subregions with a PointNet over the region’s raw points, which costs less. Both are trained with random input dropout (a per-shape drop ratio drawn from [0, 0.95]).
- Segmentation. Features are interpolated back to all points by inverse distance over the 3 nearest neighbors, with skip links from the matching abstraction level.
Results
- ModelNet40: 90.7% from 1024 xyz points vs 89.2% for PointNet; 91.9% with normals and 5000 points vs 90.1% for multi-view MVCNN. MNIST error is 0.51%, against 0.78% for PointNet.
- MSG+DP loses under 1% accuracy going from 1024 to 256 test points.
- SHREC15 non-rigid shapes: 96.09% when neighborhoods use geodesic distance on the surface and points carry intrinsic features, vs 93.03% for DeepGM and 60.18% with Euclidean xyz input.
- ShapeNet parts: 85.1 mean IoU vs 83.7 for PointNet. On ScanNet labeling (geometry only), it beats the voxel CNN and PointNet baselines, and MSG holds up best on non-uniform virtual scans.
Limitations
It is slower than PointNet. On a GTX 1080 a forward pass (batch of 8) takes 82.4 ms for SSG, 163.2 ms for MSG and 87.0 ms for MRG, against 11.6 ms for vanilla PointNet. The authors leave faster MSG and MRG layers to future work. Robustness to density was tested with random dropout and simulated virtual scans, not raw sensor scans.
Figures
Abstract
Few prior works study deep learning on point sets. PointNet by Qi et al. is a pioneer in this direction. However, by design PointNet does not capture local structures induced by the metric space points live in, limiting its ability to recognize fine-grained patterns and generalizability to complex scenes. In this work, we introduce a hierarchical neural network that applies PointNet recursively on a nested partitioning of the input point set. By exploiting metric space distances, our network is able to learn local features with increasing contextual scales. With further observation that point sets are usually sampled with varying densities, which results in greatly decreased performance for networks trained on uniform densities, we propose novel set learning layers to adaptively combine features from multiple scales. Experiments show that our network called PointNet++ is able to learn deep point set features efficiently and robustly. In particular, results significantly better than state-of-the-art have been obtained on challenging benchmarks of 3D point clouds.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.