Computer vision for robots
Features, geometry and calibration, then detection, segmentation, depth and tracking, the vision that robot perception builds on.
Start here: seminal papers
Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography★ Must read
Communications of the ACM, June 1981
RANSAC fits a model to random minimal subsets of the data and keeps the hypothesis that the most other points agree with, instead of fitting all the data and pruning the worst residuals. Applied to locating a camera from landmarks with a new closed-form three-point solver, it kept every gross error out of the final fit in 50 synthetic problems.
An Iterative Image Registration Technique with an Application to Stereo VisionSeminal
International Joint Conference on Artificial Intelligence (IJCAI), August 1981
Align two images by using the spatial intensity gradient to step the displacement toward the best match in a Newton-Raphson-style iteration, instead of trying every displacement. A converging iteration takes O(M² log N) steps on average versus O(M²N²) for exhaustive search, and the same method recovers object depths in a stereo pair.
A Computational Approach to Edge DetectionSeminal
IEEE Transactions on Pattern Analysis and Machine Intelligence, November 1986
Canny writes three goals for an edge detector (good detection, good localization, one response per edge) as functionals of the filter and optimizes them. The optimal step-edge operator is closely approximated by the first derivative of a Gaussian, about 20% below the optimum, and the full detector marks gradient maxima of the smoothed image and links them with hysteresis thresholds.
A Combined Corner and Edge DetectorSeminal
Proceedings of the Alvey Vision Conference, September 1988
Moravec's corner detector, rebuilt on the local auto-correlation matrix of image gradients, so that one response R = Det(M) - k Tr(M)^2 labels each pixel as corner, edge or flat. The paper has no numbers; on outdoor image pairs it finds corners and thin edges and links them into an edge-vertex graph.
Object Modeling by Registration of Multiple Range ImagesSeminal
IEEE International Conference on Robotics and Automation (ICRA), April 1991
Register overlapping range images by iteratively minimizing the distance from points on one view to tangent planes of the other, starting from a rough initial pose, then merge the registered views into a complete object model. On two view pairs rotated by 15 and 20 degrees, the recovered rotations were -15.06 and -19.75 degrees.
Good Features to TrackSeminal
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 1994
Pick feature windows whose 2x2 gradient matrix has a large smallest eigenvalue, because that is exactly when the tracker can solve for motion reliably, and drop features whose appearance, after an affine warp back to the first frame, stops matching. On a 26-frame real sequence with 102 features, this affine check singles out the occluded and depth-straddling features that a translation-only check misses.
Bundle Adjustment: A Modern SynthesisSeminal
Vision Algorithms: Theory and Practice (ICCV Workshop), September 1999
A survey that brings photogrammetry's bundle adjustment methods to computer vision, covering robust cost functions, sparse Newton solvers, gauge freedom and quality control. On a weakly connected 'strip' of images, sparse Gauss-Newton converges in about 6 steps, while resection-intersection fails to converge within 3000 iterations once there are 16 or more images.
A Flexible New Technique for Camera Calibration★ Must read
IEEE Transactions on Pattern Analysis and Machine Intelligence, November 2000
Calibrate a camera from a few photos of a printed flat pattern held at different orientations, using a closed-form solution from plane homographies and then a maximum-likelihood refinement with radial distortion. On a real camera, results from 2 to 5 images agree within about 2 pixels in focal length, with reprojection errors of 0.3-0.4 pixels.
Rapid Object Detection using a Boosted Cascade of Simple FeaturesSeminal
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), December 2001
A face detector that scans a cascade of AdaBoost classifiers built on simple rectangle features, computed fast with an integral image, so most image windows are rejected after a few features. It matches the best published detection rates on MIT+CMU and runs at 15 frames per second on a 700 MHz Pentium III, about 15x faster than Rowley et al.
Distinctive Image Features from Scale-Invariant Keypoints★ Must read
International Journal of Computer Vision, November 2004
SIFT detects keypoints at extrema of a difference-of-Gaussian scale space and describes each with a 128-element histogram of local gradient orientations, invariant to scale and rotation. With a nearest-neighbor ratio test, Hough clustering and an affine fit, it recognizes occluded objects in clutter from as few as 3 matches in under 0.3 s.
Histograms of Oriented Gradients for Human DetectionSeminal
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2005
Dense grids of locally contrast-normalized histograms of gradient orientations (HOG), fed to a linear SVM, detect pedestrians far better than wavelet, PCA-SIFT or shape-context features. They give near-perfect separation on the MIT set and at least an order of magnitude fewer false positives on the new, harder INRIA dataset.
Visual Servo Control, Part I: Basic ApproachesSeminal
IEEE Robotics & Automation Magazine, December 2006
A tutorial that casts visual servoing as driving a visual-feature error to zero with the velocity law v = -lambda L+ e, then compares image-based (IBVS) and position-based (PBVS) control. It reports no benchmark numbers; noise-free simulations of one positioning task show IBVS is only locally stable, while PBVS is globally stable only if the pose estimate is perfect.
Parallel Tracking and Mapping for Small AR WorkspacesSeminal
IEEE/ACM International Symposium on Mixed and Augmented Reality (ISMAR), November 2007
Track a hand-held camera in an unknown scene by splitting tracking and mapping into two parallel threads, so the map can be built from keyframes with full bundle adjustment. Frames track in about 20 ms against maps of thousands of points, and on a synthetic sequence the trajectory error is 6 mm versus 135 mm for EKF-SLAM.
KinectFusion: Real-Time Dense Surface Mapping and TrackingSeminal
IEEE International Symposium on Mixed and Augmented Reality (ISMAR), October 2011
Fuse every Kinect depth frame into one truncated signed distance volume on the GPU, and track the camera with ICP against a raycast of that model rather than the previous frame. It runs at the Kinect's 30 Hz in room-sized scenes, and the first and last poses of a 560-frame turntable loop nearly overlap with no global optimisation.
ORB: An Efficient Alternative to SIFT or SURFSeminal
IEEE International Conference on Computer Vision (ICCV), November 2011
ORB is a binary feature that adds an orientation to FAST corners and learns a rotation-aware set of BRIEF tests. It matches about as well as SIFT in their tests and takes 15.3 ms per 640x480 frame, against 217.3 ms for SURF and 5228.7 ms for SIFT.
ImageNet Classification with Deep Convolutional Neural NetworksSeminal
Advances in Neural Information Processing Systems (NeurIPS), December 2012
An eight-layer convolutional network with 60 million parameters, trained on two GPUs with ReLUs, data augmentation and dropout, classifies the 1000 ImageNet classes at 17.0% top-5 test error on ILSVRC-2010, against 25.7% for the best earlier result. An ensemble of these networks won ILSVRC-2012 with 15.3% versus 26.2% for the runner-up.
LSD-SLAM: Large-Scale Direct Monocular SLAMSeminal
European Conference on Computer Vision (ECCV), September 2014
A monocular SLAM system that uses image intensities directly instead of keypoints. It tracks the camera against keyframes with semi-dense depth maps and links the keyframes in a pose graph of sim(3) edges, so scale drift can be detected and corrected. It runs in real time on a CPU, maps hand-held outdoor trajectories of about 500 m, and reaches 1.47 cm trajectory RMSE on TUM RGB-D fr2/xyz.
U-Net: Convolutional Networks for Biomedical Image SegmentationSeminal
May 2015
A U-shaped fully convolutional network, with skip connections from a contracting to an expanding path, is trained end to end on only 20-35 annotated images using elastic-deformation augmentation. It took the best warping error on the ISBI EM segmentation challenge and won the 2015 ISBI cell tracking challenge on two light-microscopy datasets with 92% and 77.6% IoU, against 83% and 46% for the second-best entry.
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksSeminal
June 2015
A Region Proposal Network shares convolutional features with the Fast R-CNN detector, so object proposals cost about 10 ms instead of seconds. With VGG-16 the whole detector runs at 5 fps and reaches 73.2% mAP on PASCAL VOC 2007 using 300 proposals per image.
You Only Look Once: Unified, Real-Time Object DetectionSeminal
June 2015
YOLO treats object detection as a single regression, where one convolutional network sees the whole image once and outputs boxes and class probabilities on a 7x7 grid. It reaches 63.4% mAP on VOC 2007 at 45 fps, and the smaller Fast YOLO reaches 52.7% at 155 fps, twice the mAP of earlier real-time detectors.
Deep Residual Learning for Image Recognition★ Must read
December 2015
Let each few layers learn a residual F(x) that an identity shortcut adds back to the input, which makes very deep networks trainable. A 152-layer ResNet reaches 4.49% single-model top-5 error on ImageNet validation, and an ensemble gets 3.57% on the test set, 1st place in ILSVRC 2015.
Structure-from-Motion RevisitedSeminal
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016
COLMAP, an open-source incremental Structure-from-Motion pipeline with new geometric verification, next-best-view selection, robust triangulation and iterative bundle adjustment. It registers the most images on all 17 Internet photo collections tested, e.g. 20,918 of 74,394 Rome photos versus 14,797 for VisualSFM.
ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras★ Must read
October 2016
ORB-SLAM2 extends the feature-based monocular ORB-SLAM to stereo and RGB-D cameras, with loop closing, relocalization and map reuse running in real time on a CPU. It is the most accurate of the compared SLAM systems on most KITTI, EuRoC and TUM RGB-D sequences, with relative drift generally under 1% on KITTI.
Mask R-CNNSeminal
March 2017
Add a mask-predicting branch and a quantization-free RoIAlign layer to Faster R-CNN to get instance segmentation at 5 fps. Mask R-CNN reaches 37.1 mask AP on COCO test-dev, above the 2016 challenge winner, and the same framework also detects boxes and human keypoints.
Open3D: A Modern Library for 3D Data ProcessingSeminal
January 2018
Open3D is an open-source C++11 library with Python bindings for point clouds, meshes and RGB-D images, covering I/O, visualization, registration and volumetric reconstruction. Its ICP is up to 25x faster than PCL's, and its scene reconstruction pipeline up to an order of magnitude faster than the original implementation.
Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset TransferSeminal
July 2019
Train one monocular depth network on five datasets with incompatible ground truth, including a new dataset mined from 3D movies, using a scale- and shift-invariant loss and Pareto-optimal dataset mixing. Tested zero-shot on six unseen datasets, every prior method is at least 65% worse on average.
RAFT: Recurrent All-Pairs Field Transforms for Optical FlowSeminal
March 2020
RAFT estimates optical flow by correlating every pixel with every other pixel and refining a single high-resolution flow field with a small weight-tied recurrent unit. It cuts the best published error by 30% on Sintel final (2.855 px EPE) and by 16% on KITTI (5.10% F1-all).
An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleSeminal
October 2020
Cut an image into 16x16 patches and feed them, like word tokens, to a standard Transformer encoder. Pre-trained on 300M images, ViT-H/14 reaches 88.55% on ImageNet and 77.63% on VTAB with about a quarter of the pre-training compute of the best ResNet.
Learning Transferable Visual Models From Natural Language Supervision★ Must read
February 2021
Train an image encoder and a text encoder on 400 million web (image, text) pairs to tell which caption goes with which image, then classify any dataset zero-shot by embedding its class names. CLIP reaches 76.2% zero-shot accuracy on ImageNet, matching the original ResNet-50 without using any of its labeled training images.
Emerging Properties in Self-Supervised Vision TransformersSeminal
April 2021
DINO trains a Vision Transformer without labels by having a student network match a momentum teacher's output on different crops of the same image. The [CLS] token's attention maps segment objects on their own, and the features reach 80.1% ImageNet top-1 with a linear classifier and 78.3% with plain k-NN.
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object DetectionSeminal
March 2023
Grounding DINO turns the Transformer detector DINO into an open-set detector that finds any object named in a text prompt, by fusing language and image features in the neck, the query initialization and the decoder. It reaches 52.5 AP on COCO without any COCO training data and a zero-shot record of 26.1 mean AP on ODinW.
Segment Anything★ Must read
April 2023
SAM is a promptable segmentation model trained on SA-1B, 1.1B automatically generated masks on 11M images, and applied to new tasks zero-shot by prompting. From a single point it beats the strongest interactive segmenter, RITM, on 16 of 23 unseen datasets.
DINOv2: Learning Robust Visual Features without SupervisionSeminal
April 2023
A 1.1B-parameter ViT trained with self-supervision on 142M automatically curated images, then distilled into smaller models. Its frozen features reach 86.5% ImageNet-1k linear-probe accuracy, slightly ahead of OpenCLIP, and beat prior self-supervised features on segmentation, depth and retrieval.
CoTracker: It is Better to Track TogetherSeminal
July 2023
A transformer point tracker that tracks many 2D points in a video jointly, with attention between tracks, instead of one at a time. Trained only on synthetic Kubric, it raises Average Jaccard on TAP-Vid-DAVIS from 56.2 (TAPIR) to 62.2.
Depth Anything: Unleashing the Power of Large-Scale Unlabeled DataSeminal
January 2024
A monocular relative-depth model trained on 1.5M labeled images plus 62M unlabeled images pseudo-labeled by a teacher, with strong perturbations on the student and a feature-alignment loss to frozen DINOv2. Zero-shot on six unseen datasets it beats MiDaS v3.1 (KITTI AbsRel 0.076 vs 0.127), and fine-tuned it reaches 0.984 delta1 on NYUv2.
SAM 2: Segment Anything in Images and VideosSeminal
August 2024
SAM 2 extends Segment Anything to video with a streaming memory that carries an object's mask from frame to frame, trained on a new 35.5M-mask video dataset collected with the model in the loop. It beats SAM-plus-tracker baselines with over 3x fewer interactions, and on images it is more accurate than SAM and 6x faster.
Recent
Semantics-to-Contact: A Stagewise Framework for Robust Contact-Rich Manipulation
November 2026
S2C uses a vision-language model to bring the robot near the contact point, then learns the final force-aware insertion with human-in-the-loop residual RL on the real robot, trained on wrist images with randomized backgrounds. Under new tablecloths and lighting it averages 69-84% success on six tasks, while HIL-SERL and HIL-ConRFT mostly fail.
AnchorHEC: Targetless Hand-Eye Calibration of Wrist and Fixed Cameras
November 2026
AnchorHEC calibrates a robot's wrist camera and fixed cameras in one run from ordinary RGB images of the scene and the arm's forward kinematics, with no calibration board. On a physical xArm6 with 15 wrist poses it reaches 1.2-1.4 cm wrist-camera and 1.6-1.8 cm fixed-camera translation error, versus 2.4-10.2 cm and 3.4-8.8 cm for the JCR baseline.
New papers stay for 14 days.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.