PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Charles R. Qi, Hao Su, Kaichun Mo, Leonidas J. Guibas

★ Must read December 2016

TL;DR A neural network that consumes raw, unordered point clouds directly, using a shared per-point MLP followed by max pooling. One architecture handles object classification (89.2% on ModelNet40), part segmentation and scene segmentation, at 141x fewer FLOPs than a multi-view CNN.

What they do

Deep networks for 3D data usually converted point clouds to voxel grids or rendered images first. PointNet takes the point set itself as input and stays invariant to point order by max-pooling over features computed independently for each point. The same backbone performs shape classification, part segmentation and room-scale semantic segmentation.

How it works

Per-point features plus max pooling. A shared MLP lifts each (x,y,z) point to 1024 dimensions, and max pooling across points gives a global shape feature.
Segmentation. The global feature is concatenated back onto each point’s feature, and a second MLP predicts per-point labels.
Alignment networks. Mini-PointNets (T-nets) predict a 3x3 transform for the input points and a 64x64 transform for the features. A regularizer keeps the feature transform close to orthogonal.
Theory. The network can approximate any continuous set function. Its output is determined by a “critical point set” of at most K points, where K is the size of the max-pooled feature.

Results

ModelNet40: 89.2% overall accuracy from one view, equal to the 20-view volumetric Subvolume model. Multi-view MVCNN reaches a higher 90.1% average class accuracy, but PointNet needs 141x fewer FLOPs and 17x fewer parameters.
ShapeNet part segmentation: 83.7 mIoU, against 81.4 for the best prior method. Simulated partial Kinect scans cost 5.3 mIoU.
Stanford indoor scans: 47.71 mIoU against 20.12 for a hand-crafted-feature baseline. A detector built on the segmentation reaches 24.24 mean AP against 18.22 for prior work.
Robustness: with 50% of points missing, accuracy drops 3.7%, while VoxNet drops from 86.3% to 46.0%. Max pooling beats average pooling (87.1% vs 83.8%) and an LSTM (78.5%).

Limitations

The authors attribute the remaining gap to multi-view CNNs to lost fine geometric detail. Each point is encoded independently, so local context enters only through the global feature. Robustness to outliers holds only when the network saw outliers in training. The scene detector’s connected components fail on tightly packed chairs.

Figures

From Figure 1: One architecture that takes a raw point cloud, with no voxelization or rendering, for classification, ... — From Figure 1: One architecture that takes a raw point cloud, with no voxelization or rendering, for classification, part segmentation and scene semantic segmentation.

From Figure 2: The classification network transforms n input points, applies a shared MLP to each point and max-pools... — From Figure 2: The classification network transforms n input points, applies a shared MLP to each point and max-pools them into a global feature. The segmentation network concatenates that global feature with each point feature and outputs per-point scores.

From Figure 4: Semantic segmentation of scanned rooms. Top row is the colored input point cloud, bottom row the predi... — From Figure 4: Semantic segmentation of scanned rooms. Top row is the colored input point cloud, bottom row the predicted labels from the same viewpoint.

From Figure 7: Critical points and upper-bound shapes. Any point cloud between the critical point set and the upper-b... — From Figure 7: Critical points and upper-bound shapes. Any point cloud between the critical point set and the upper-bound shape gives exactly the same global feature. Color shows depth.

Abstract

Point cloud is an important type of geometric data structure. Due to its irregular format, most researchers transform such data to regular 3D voxel grids or collections of images. This, however, renders data unnecessarily voluminous and causes issues. In this paper, we design a novel type of neural network that directly consumes point clouds and well respects the permutation invariance of points in the input. Our network, named PointNet, provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing. Though simple, PointNet is highly efficient and effective. Empirically, it shows strong performance on par or even better than state of the art. Theoretically, we provide analysis towards understanding of what the network has learnt and why the network is robust with respect to input perturbation and corruption.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong