← Paper Digest

On the Utility of Pre-Trained Visual Representations for Policy Learning of Contact-Rich Insertion Tasks

Leon Kim, Dinesh Jayaraman, Michael Posa

November 2026

TL;DR A controlled study of which visual features help diffusion policies with tight insertion, comparing DINOv2, Theia, segmentation masks and a new contact-centric map in simulation and on a real book-insertion task. The best feature changes even between near-identical tasks, and only ground-truth contact information helped in every simulated setting.

What they do

Studies of pre-trained visual features for robot policies span many tasks and reach conflicting conclusions. This paper narrows the question to tight insertion. It adds pre-trained or hand-designed features to the input of a diffusion policy and measures success on two closely related insertion tasks. Each task is perturbed in tolerance or camera viewpoint, and the book task is repeated on a real robot.

How it works

  • Policy. A diffusion policy with a ResNet-18 encoder sees RGB-D, the end-effector pose and recent actions. It predicts 1.4 s of SE(3) end-effector poses at 5 Hz, which a 1 kHz impedance controller tracks.
  • Features compared. DINOv2 and Theia patch features, segmentation masks of the grasped object (ground truth or SAM-2), and a new contact-centric map. That map is estimated from depth and marks near-contact regions with distance-to-contact and surface normals. It is trained on a small synthetic dataset.
  • Tasks. In ManiSkill: peg-in-hole at 3, 9 and 15 mm tolerance, and book insertion into a slot 3.5 mm too narrow. Book insertion is also tested with the camera rotated to more oblique views. On the real robot, book insertion uses 288 demonstrations over 6 books and is tested with 10 attempts on each of 3 unseen books.

Results

  • In simulation, neither DINOv2 nor Theia beat training from scratch. On the real robot, Theia gave significant gains at both data budgets (4 and 6 training books).
  • Masks helped a lot in simulated book insertion but not in peg-in-hole, where the peg is occluded.
  • Ground-truth contact maps were the only feature to significantly beat the baseline at every peg tolerance. Margins were largest at 30-50 demonstrations and 3-9 mm tolerance.
  • The estimated contact map matched ground truth on simulated books, but gave no gain on pegs or, on its own, on the real robot. Combined with masks it beat the baseline at both real data budgets.

Limitations

The real-world evaluation is small: one task, 30 trials per policy and a fixed slot location. Results vary with which demonstrations are used for training. The contact estimator is a proof of concept that degrades under occlusion and sensor noise.

Figures

From Figure 1: The real-world book-insertion task, with the contact-centric feature map overlaid on the camera frame.
From Figure 1: The real-world book-insertion task, with the contact-centric feature map overlaid on the camera frame.
From Figure 4: Simulated book insertion. The slot is narrower than the book, so the robot pries the neighboring books...
From Figure 4: Simulated book insertion. The slot is narrower than the book, so the robot pries the neighboring books apart.
From Figure 8: Success rate on simulated book insertion against the number of demonstrations; error bars are standard...
From Figure 8: Success rate on simulated book insertion against the number of demonstrations; error bars are standard error over 5 random demonstration subsets.
From Figure 10: Real-world book-insertion success with 4 or 6 training books; error bars are 95% confidence bounds ov...
From Figure 10: Real-world book-insertion success with 4 or 6 training books; error bars are 95% confidence bounds over 30 episodes.
Abstract

Visuomotor imitation learning has become a predominant paradigm for real-world robotic manipulation, yet it remains data-inefficient and brittle to task-irrelevant visual shifts. Pre-trained visual representations promise to ease this burden by distilling priors from internet-scale data, but the ``right'' representation for robotics remains unresolved: existing studies span broad but sparse task distributions and reach conclusions that vary and even conflict, pointing to strong task and setting dependence. We examine this question by bringing a magnifying glass to a common contact-rich manipulation task: insertion. We compare representations from widely used vision foundation models against an explicit contact-centric representation on two closely related insertion tasks, perturb each along controlled axes of occlusion and physical tolerance, and test whether conclusions transfer to the real world. Task dependence proves to be more subtle than previous, broadly scoped evaluations revealed: the best representation differs even between our near-identical tasks, and small changes shift trends in unpredictable ways. Despite this variability, results across simulation and real-world experiments demonstrate the distinct value of spatially-grounded, and in particular, contact-centric information for contact-rich policy learning, while underscoring how difficult it is to estimate such cues reliably under realistic visual complexity and sensor noise. We hope these findings motivate further study of representations in contact-rich manipulation, along with advances in contact perception and its integration into visuomotor policy learning.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.