3D perception and scene representation
Object pose tracking, Gaussian splatting and building simulations from the real world.
Start here: seminal papers
Object Modeling by Registration of Multiple Range ImagesSeminal
IEEE International Conference on Robotics and Automation (ICRA), April 1991
Register overlapping range images by iteratively minimizing the distance from points on one view to tangent planes of the other, starting from a rough initial pose, then merge the registered views into a complete object model. On two view pairs rotated by 15 and 20 degrees, the recovered rotations were -15.06 and -19.75 degrees.
Bundle Adjustment: A Modern SynthesisSeminal
Vision Algorithms: Theory and Practice (ICCV Workshop), September 1999
A survey that brings photogrammetry's bundle adjustment methods to computer vision, covering robust cost functions, sparse Newton solvers, gauge freedom and quality control. On a weakly connected 'strip' of images, sparse Gauss-Newton converges in about 6 steps, while resection-intersection fails to converge within 3000 iterations once there are 16 or more images.
FastSLAM: A Factored Solution to the Simultaneous Localization and Mapping ProblemSeminal
AAAI Conference on Artificial Intelligence, July 2002
FastSLAM splits SLAM into a particle filter over robot paths and, inside each particle, a small Kalman filter for each landmark. A shared tree structure brings each update down to O(M log K) time, and the method mapped 50,000 simulated landmarks with 100 particles.
The GraphSLAM Algorithm with Applications to Large-Scale Mapping of Urban StructuresSeminal
The International Journal of Robotics Research, May 2006
GraphSLAM turns every odometry reading and measurement into a soft constraint in a sparse graph, eliminates the map features to leave a smaller problem over robot poses, and solves it. It builds maps with 10^8 or more features; relaxing the largest urban data set (600 m by 800 m, a dozen loops) takes about 30 seconds.
Parallel Tracking and Mapping for Small AR WorkspacesSeminal
IEEE/ACM International Symposium on Mixed and Augmented Reality (ISMAR), November 2007
Track a hand-held camera in an unknown scene by splitting tracking and mapping into two parallel threads, so the map can be built from keyframes with full bundle adjustment. Frames track in about 20 ms against maps of thousands of points, and on a synthetic sequence the trajectory error is 6 mm versus 135 mm for EKF-SLAM.
iSAM: Incremental Smoothing and MappingSeminal
IEEE Transactions on Robotics, December 2008
iSAM solves SLAM as sparse least squares and updates the QR factor of the smoothing problem with Givens rotations as measurements arrive, reordering variables periodically to keep it sparse. On the 26-minute Victoria Park run it computes the full, exact solution with data association after every frame in 7.7 minutes, over 3x faster than real time.
g2o: A General Framework for Graph OptimizationSeminal
IEEE International Conference on Robotics and Automation (ICRA), May 2011
An open-source C++ framework that optimizes any least-squares problem that can be written as a graph, such as 2D/3D SLAM and bundle adjustment, once the user defines an error function and an update operator. On ten datasets it is faster per iteration than √SAM on every SLAM dataset, on par with problem-specific solvers elsewhere, and about twice as fast as RobotVision.
KinectFusion: Real-Time Dense Surface Mapping and TrackingSeminal
IEEE International Symposium on Mixed and Augmented Reality (ISMAR), October 2011
Fuse every Kinect depth frame into one truncated signed distance volume on the GPU, and track the camera with ICP against a raycast of that model rather than the previous frame. It runs at the Kinect's 30 Hz in room-sized scenes, and the first and last poses of a 560-frame turntable loop nearly overlap with no global optimisation.
Occupancy Grids: A Stochastic Spatial Representation for Active Robot PerceptionSeminal
March 2013
Represent the robot's surroundings as a grid of cells, each holding a Bayesian estimate of the probability that it is occupied, updated from noisy range readings taken from many views and sensors. The paper reviews the framework and shows sonar and sonar-plus-stereo maps, path planning and pose-uncertainty handling done directly on the grid, without quantitative metrics.
Estimating Uncertain Spatial Relationships in RoboticsSeminal
March 2013
Stack the poses of the robot and every mapped object into one state vector with a full covariance matrix (the stochastic map), and build and update it with pose compounding, inversion and an extended Kalman filter. The paper cites Monte Carlo tests in which these linearized estimates stayed within 1% of the true means and variances for angular errors up to 5 degrees.
LSD-SLAM: Large-Scale Direct Monocular SLAMSeminal
European Conference on Computer Vision (ECCV), September 2014
A monocular SLAM system that uses image intensities directly instead of keypoints. It tracks the camera against keyframes with semi-dense depth maps and links the keyframes in a pose graph of sim(3) edges, so scale drift can be detected and corrected. It runs in real time on a CPU, maps hand-held outdoor trajectories of about 500 m, and reaches 1.47 cm trajectory RMSE on TUM RGB-D fr2/xyz.
Revisiting Active PerceptionSeminal
March 2016
A history and position paper that defines an active perceiver as an agent that knows why it wants to sense, then chooses what, how, when and where to sense, and organizes 45+ years of work around those five questions. It runs no new experiments and argues that a complete artificial agent must perceive actively.
Structure-from-Motion RevisitedSeminal
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016
COLMAP, an open-source incremental Structure-from-Motion pipeline with new geometric verification, next-best-view selection, robust triangulation and iterative bundle adjustment. It registers the most images on all 17 Internet photo collections tested, e.g. 20,918 of 74,394 Rome photos versus 14,797 for VisualSFM.
ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras★ Must read
October 2016
ORB-SLAM2 extends the feature-based monocular ORB-SLAM to stereo and RGB-D cameras, with loop closing, relocalization and map reuse running in real time on a CPU. It is the most accurate of the compared SLAM systems on most KITTI, EuRoC and TUM RGB-D sequences, with relative drift generally under 1% on KITTI.
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation★ Must read
December 2016
A neural network that consumes raw, unordered point clouds directly, using a shared per-point MLP followed by max pooling. One architecture handles object classification (89.2% on ModelNet40), part segmentation and scene segmentation, at 141x fewer FLOPs than a multi-view CNN.
Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics★ Must read
March 2017
Train a CNN to score parallel-jaw grasps from depth images using only synthetic data, 6.7 million rendered point clouds labeled with a robust analytic grasp metric. On an ABB YuMi it plans grasps in 0.8 s with 93% success on known adversarial objects and reaches 99% precision on 40 novel household objects.
PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric SpaceSeminal
June 2017
PointNet++ applies PointNet recursively to nested local neighborhoods of a point cloud, so it learns local features at growing scales, and its multi-scale layers trained with random point dropout cope with uneven sampling density. It reaches 90.7% on ModelNet40 from 1024 xyz points, against 89.2% for PointNet.
Grasp Pose Detection in Point CloudsSeminal
June 2017
Sample 6-DOF parallel-jaw grasp candidates on any visible surface of a point cloud and score each with a small CNN over multi-view projections of the gripper's closing region. A Baxter clearing dense clutter of novel objects succeeds on 93% of 288 grasp attempts.
PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered ScenesSeminal
November 2017
PoseCNN labels each pixel, lets pixels vote for each object's 2D center to recover its 3D translation, and regresses rotation as a quaternion, with a loss that tolerates symmetric objects. With ICP refinement it reaches 78.0% mean accuracy on OccludedLINEMOD, and it comes with the new YCB-Video dataset of 133,827 annotated frames.
Open3D: A Modern Library for 3D Data ProcessingSeminal
January 2018
Open3D is an open-source C++11 library with Python bindings for point clouds, meshes and RGB-D images, covering I/O, visualization, registration and volumetric reconstruction. Its ICP is up to 25x faster than PCL's, and its scene reconstruction pipeline up to an order of magnitude faster than the original implementation.
DeepIM: Deep Iterative Matching for 6D Pose EstimationSeminal
March 2018
DeepIM refines an initial 6D object pose from one RGB image by repeatedly rendering the object's 3D model and predicting the pose correction that aligns the rendering with the image. Starting from PoseCNN, it raises LINEMOD 6D Pose accuracy from 62.7% to 88.6% and also refines objects never seen in training.
Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic ManipulationSeminal
June 2018
A network trained by self-supervision on robot-collected RGBD scans gives every pixel a descriptor. The descriptor for a given point stays the same across views and deformations, and optionally across instances of a class. It trains in about 20 minutes per object (47 objects tested) and lets a robot grasp a point that a user clicked in a single reference image.
PointNetGPD: Detecting Grasp Configurations from Point SetsSeminal
September 2018
PointNetGPD scores parallel-jaw grasp candidates by running a PointNet classifier on the raw points inside the gripper, trained on 350k real point clouds and grasps over YCB objects. It beats the GPD baseline at grasp-quality classification (84.75% vs 79.96% on single-view clouds), and on a real UR5 it clears more of each cluttered pile (completion 95-100% vs 81.5-95%).
Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size EstimationSeminal
January 2019
A CNN predicts, for each pixel of an unseen object, its coordinates in a normalized space shared by the whole category (NOCS), and aligning these with the depth map gives the object's metric 6D pose and size. On real test scenes it reaches 76.4% mAP at 3D IoU 50 and 23.1% at (10°, 5 cm), against 43.8% and 0.8% for a Mask R-CNN + ICP baseline.
DenseFusion: 6D Object Pose Estimation by Iterative Dense FusionSeminal
January 2019
DenseFusion estimates the 6D pose of known objects from RGB-D by fusing color and point-cloud features at every pixel, letting each pixel vote for a pose, and refining it with a learned network instead of ICP. On YCB-Video, 96.8% of its poses fall under 2 cm ADD-S error, versus 93.2% for PoseCNN+ICP, and it runs about 200x faster.
PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose TrackingSeminal
May 2019
A particle filter tracks an object's 6D pose in video by sampling translations and keeping a full rotation distribution for each particle, scored by matching auto-encoder embeddings against a precomputed codebook. It captures symmetry ambiguity without symmetry labels and doubles RGB recall on T-LESS over single-frame estimation.
6-DOF GraspNet: Variational Grasp Generation for Object ManipulationSeminal
May 2019
A variational autoencoder samples diverse 6-DOF grasps from a partial object point cloud, and a learned evaluator scores them and refines them by gradient ascent. Trained only on simulated grasps, it picks up 17 real objects with 88% success versus 47% for GPD.
Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset TransferSeminal
July 2019
Train one monocular depth network on five datasets with incompatible ground truth, including a new dataset mined from 3D movies, using a scale- and shift-invariant loss and Pareto-optimal dataset mixing. Tested zero-shot on six unseen datasets, every prior method is at least 65% worse on average.
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis★ Must read
March 2020
An MLP maps a 3D point and viewing direction to density and color, and volume rendering along camera rays turns it into images, so one network per scene is fit from posed photos alone. On realistic synthetic objects NeRF reaches 31.01 PSNR, against 26.05 for the best prior method.
3D Shape Reconstruction from Vision and TouchSeminal
July 2020
Reconstructs an object's 3D mesh from one RGB image plus four simulated high-resolution touch readings from a grasping robot hand, by deforming mesh patches from vision around exact patches from touch. Adding touch lowers test Chamfer distance from 1.074 to 0.991 with the hand occluding the view, and five grasps beat one (0.654 vs 0.804).
iNeRF: Inverting Neural Radiance Fields for Pose EstimationSeminal
December 2020
iNeRF recovers a camera's 6DoF pose from one RGB image by gradient descent on the pixel error between the image and renderings from a trained NeRF, with no mesh model. On unseen ShapeNet cars, using a NeRF predicted from a single view, it reaches 4.39° mean rotation error against 9.27° for a SuperGlue baseline.
Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in ClutterSeminal
January 2021
A 3D fully convolutional network takes a TSDF of a cluttered scene and predicts grasp quality, gripper orientation and opening width at every voxel in one 10 ms forward pass. Trained only on simulated grasps, it clears 92% of objects in real-robot clutter removal without explicit collision checking.
iMAP: Implicit Mapping and Positioning in Real-TimeSeminal
March 2021
A real-time RGB-D SLAM system whose only map is a small MLP trained live from scratch, tracking at 10 Hz and mapping at 2 Hz. On Replica it fills in more of the room than TSDF fusion (79% vs 75% completion ratio) with about 13 keyframes and 60x less memory.
Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes★ Must read
March 2021
A single PointNet++ network treats each observed depth point as a possible grasp contact and predicts a full 6-DoF parallel-jaw grasp there, so it only has to learn 4 DoF. Trained on simulated grasps only, it picks unseen objects from cluttered scenes on a Franka with 90.2% success, versus 80.4% for the prior best pipeline.
Vision-Only Robot Navigation in a Neural Radiance WorldSeminal
October 2021
A robot plans through a pre-trained NeRF by minimizing NeRF density along a differentially flat trajectory and localizes by matching its RGB images to NeRF renders, replanning in a loop. In simulated scenes a quadrotor reaches the goal without collision where open-loop execution crashes, but each 0.1 s step still takes about 2 s to replan and 4 s to estimate state.
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for ManipulationSeminal
December 2021
Describe an object by an SE(3)-equivariant neural field of 3D descriptors, learned by self-supervised shape reconstruction, and transfer grasp and place poses from 10 demonstrations by matching those descriptors. In simulation, mug pick-and-place on unseen mugs succeeds 88% of the time upright and 58% in arbitrary poses, against 45% and 17% for a 2D-descriptor baseline.
Perceiver-Actor: A Multi-Task Transformer for Robotic ManipulationSeminal
September 2022
PerAct encodes a language goal and a voxelized RGB-D scene with a Perceiver Transformer and picks the next gripper pose by classifying voxels. One agent trained on 18 RLBench tasks beats a 3D ConvNet baseline by 2.83x with 100 demos per task, and a real Franka learns 7 tasks from 53 demos.
ActiveNeRF: Learning where to See with Uncertainty EstimationSeminal
September 2022
ActiveNeRF makes NeRF predict a variance for each point's color, then adds the new views that would shrink that uncertainty the most to a small training set. With 4 starting views plus 16 chosen ones on synthetic scenes, it reaches 26.24 PSNR versus 24.25 for random views and 26.00 for furthest-view selection.
MidasTouch: Monte-Carlo inference over distributions across sliding touchSeminal
October 2022
A particle filter localizes a DIGIT touch sensor sliding over a known object by matching learned codes of local 3D tactile geometry against a precomputed per-object codebook. Over 500 real trials on 10 YCB objects, the final pose error averages 1.97 cm and 21.48°.
MegaPose: 6D Pose Estimation of Novel Objects via Render & CompareSeminal
December 2022
A render-and-compare pose estimator trained once on 2 million synthetic images of over 20K objects, then applied without retraining to novel objects given only their CAD model and a 2D detection. Its RGB-D refiner adds +41.0 mean AR over its coarse estimates across the 7 BOP datasets, competitive with methods trained on the test objects.
LERF: Language Embedded Radiance FieldsSeminal
March 2023
LERF distills multi-scale CLIP embeddings into a NeRF, so a scene captured with a phone can be queried with free-form text and returns 3D relevancy maps in real time. It localizes 80.3% of 72 long-tail objects across 5 scenes, against 54.8% for OWL-ViT and 18.0% for LSeg distilled into 3D.
BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown ObjectsSeminal
March 2023
BundleSDF tracks the 6-DoF pose of an unknown rigid object in an RGBD video and reconstructs its textured mesh at the same time, given only a mask in the first frame. On the HO3D hand-object benchmark it reaches 92.6% ADD versus 66.0% for the next best method, with 0.57 cm chamfer error.
Distilled Feature Fields Enable Few-Shot Language-Guided ManipulationSeminal
July 2023
F3RM distills DINO or CLIP image features into a NeRF of the tabletop and finds 6-DOF grasps and placements on unseen objects from two demonstrations per task or from a text query. CLIP ResNet features succeed in 39/50 grasp and place trials, and language-guided grasping succeeds on 31/50 queries.
3D Gaussian Splatting for Real-Time Radiance Field Rendering★ Must read
August 2023
Represent a scene as millions of anisotropic 3D Gaussians, start them from SfM points, and render them with a tile-based GPU rasterizer. On the Mip-NeRF360 scenes it nearly matches Mip-NeRF360 quality (PSNR 27.21 vs 27.69) after about 41 minutes of training instead of 48 hours, and renders at 134 fps instead of 0.06 fps.
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and PlanningSeminal
September 2023
Build an open-vocabulary 3D scene graph by fusing SAM masks and CLIP features across RGB-D views, captioning each object with LLaVA and GPT-4 and letting GPT-4 label the edges, with no training. Human raters judge about 70% of node captions and 90% of edges correct, and GPT-4 planning over the graph answers negation queries far better than CLIP (R@1 0.80 vs 0.26 on Replica).
FisherRF: Active View Selection and Uncertainty Quantification for Radiance Fields using Fisher InformationSeminal
November 2023
FisherRF scores candidate camera poses by the expected information gain computed from the Fisher Information of a radiance field's parameters, without needing the images at those poses. With 10 views on Blender and 3D Gaussian Splatting it reaches 23.68 dB PSNR, against 22.98 dB for ActiveNeRF-style selection and 22.49 dB for random views.
SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAMSeminal
December 2023
SplaTAM is a dense RGB-D SLAM system that uses 3D Gaussian Splatting as its map, tracking the camera and growing the map by differentiable rendering. It tracks ScanNet++ with 1.2 cm trajectory error where Point-SLAM and ORB-SLAM3 fail, and cuts Replica error from 0.52 cm to 0.36 cm.
FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects★ Must read
December 2023
One RGBD network estimates and tracks the 6D pose of objects it has never seen, given either a CAD model or about 16 reference images, with no fine-tuning. Trained only on synthetic data, it reaches 83.3 mean AR on three BOP datasets, above the instance-level SurfEmb + ICP at 79.7.
Neural feels with neural fields: Visuo-tactile perception for in-hand manipulationSeminal
December 2023
NeuralFeels fuses a camera, DIGIT touch sensors on four fingertips and joint angles to learn an object's neural SDF online and track its pose while an Allegro hand rotates it. Across 70 simulated and real trials it reaches an 81% reconstruction F-score with 4.7 mm average pose drift (2.3 mm when the CAD model is given).
DUSt3R: Geometric 3D Vision Made Easy★ Must read
December 2023
A transformer takes two images with unknown cameras and regresses a 3D point for every pixel of both, in one shared frame; a global alignment step merges many pairs into a full scene. With no task-specific training it reaches 96.2 RRA@15 on CO3Dv2 multi-view pose, against 80.5 for PoseDiffusion.
Depth Anything: Unleashing the Power of Large-Scale Unlabeled DataSeminal
January 2024
A monocular relative-depth model trained on 1.5M labeled images plus 62M unlabeled images pseudo-labeled by a teacher, with strong perturbations on the student and a feature-alignment loss to frozen DINOv2. Zero-shot on six unseen datasets it beats MiDaS v3.1 (KITTI AbsRel 0.076 vs 0.127), and fine-tuned it reaches 0.984 delta1 on NYUv2.
Splat-Nav: Safe Real-Time Robot Navigation in Gaussian Splatting MapsSeminal
March 2024
Plans and localizes a drone directly in a Gaussian Splatting map. Polytope corridors built from the splat's ellipsoids give provably safe Bézier trajectories, and RGB-only PnP against splat renders gives the pose. Re-planning runs at over 2 Hz and localization at about 25 Hz, and only one of 120 hardware flights collided.
Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust ManipulationSeminal
March 2024
RialTo scans a real scene into an articulated simulation, carries about 15 real demos into it, fine-tunes the imitation policy there with sparse-reward RL, and distills it back to a point-cloud policy. On real tasks it averages 91%, 77% and 75% success under pose randomization, distractors and disturbances, against 25%, 11% and 5% for imitation learning.
3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D RepresentationsSeminal
March 2024
DP3 conditions a diffusion policy on a 64-dim feature from a small MLP encoder over a sparse, colorless point cloud from a single camera. Across 72 simulation tasks it averages 74.4% success versus 59.8% for image-based Diffusion Policy, and it reaches 85% on 4 real tasks from 40 demonstrations each.
SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian SplattingSeminal
September 2024
SplatSim swaps a simulator's mesh renderer for a Gaussian Splat of the real scene, moving each robot link and object's Gaussians to follow the PyBullet state. RGB diffusion policies trained only on these renderings transfer zero-shot to a real UR5 with 86.25% average success over four tasks, against 97.5% for policies trained on real data.
VGGT: Visual Geometry Grounded TransformerSeminal
March 2025
VGGT is a 1.2B-parameter transformer that takes one to hundreds of images of a scene and predicts cameras, depth maps, point maps and point tracks in a single forward pass. Feed-forward in about 0.2 s, it beats optimization-based methods on camera pose, e.g. 85.3 vs 78.9 AUC@30 for VGGSfM v2 (about 10 s) on unseen RealEstate10K.
Recent
MOMASPLAT: Dynamic Novel-View Demonstration Augmentation for Mobile Manipulation
November 2026
MoMaSplat rebuilds the scene and the manipulated object as separate Gaussian splats from a mobile robot's own data, then re-renders whole demonstrations, including the object moving, from new base poses. On five real TIAGo tasks it succeeds on 37% of rollouts from unseen base poses, against 17% for the best baseline.
TwinBridge: Digital Twin-Driven Policy and Reward Transfer for Real-World Reinforcement Learning in Laboratory Robotics
November 2026
TwinBridge builds Gaussian-splatting digital twins of lab benches, pretrains a policy in them, and distills simulator-derived progress labels into a visual reward model for human-in-the-loop RL on the real robot. Across five lab tasks it reaches 98% mean autonomous success with at most 25 minutes of real-world interaction per task.
New papers stay for 14 days.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.