Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Semantics-to-Contact: A Stagewise Framework for Robust Contact-Rich Manipulation

Guanren Qiao, Ruixiang Ouyang, Sheng Xu, Yueci Deng, Yunxin Tai, Kui Jia, Guiliang Liu

November 2026

Real-robot insertion and hard-drive removal runs, four views side by side.

TL;DR S2C uses a vision-language model to bring the robot near the contact point, then learns the final force-aware insertion with human-in-the-loop residual RL on the real robot, trained on wrist images with randomized backgrounds. Under new tablecloths and lighting it averages 69-84% success on six tasks, while HIL-SERL and HIL-ConRFT mostly fail.

What they do

Real-world RL can learn precise insertions directly on hardware, but image-based policies overfit to the look of the training scene. S2C splits each task in two. A vision-language model finds where contact should happen, and motion planning brings the robot there. Then a human-in-the-loop residual RL policy learns the final force-aware contact. Randomizing the background of the wrist-camera images keeps that policy focused on the robot and the objects.

How it works

  • Semantic focusing. MolmoPoint-8B gets a third-person image and an instruction (“Point to the square USB port on the white charger”) and returns a 2D point. The depth map and hand-eye calibration turn it into a 3D target in the robot frame.
  • Contact refinement. An ACT policy trained on demonstrations is the base. A residual policy trained with SAC adds corrections from two wrist cameras, end-effector pose and velocity, and force-torque. An operator can intervene with a SpaceMouse, as in HIL-SERL.
  • Object-centric augmentation. Prompted with names of the robot, gripper and objects, YOLOE segments each wrist image. The background is replaced with random colors and textures, and the foreground gets color and brightness jitter.

Results

  • Training scene with shifted targets: 94.17% average success over six tasks (charger, USB and test-tube insertion), vs 90.83% for HIL-SERL and 91.76% for HIL-ConRFT. ACT, Octo-small and SERL reach at most 5.83%.
  • With a patterned tablecloth, strong sunlight or colored light, both RL baselines score 0/20 on every task, while S2C averages 69.17%, 84.17% and 79.17%.
  • Ablation on two tasks: without augmentation, success falls from 16/20 and 14/20 to 0/20. Without VLM guidance it is 10/20 and 12/20, and without the human in the loop it is zero.

Limitations

S2C fails when the contact region itself changes: 0/10 on charger insertion with unfamiliar objects inside the power strip, and 0/10 when the white charger is swapped for a black one. Training needs an operator, 30-40 demonstrations per task and a calibrated third-person camera. Only short insertion and removal skills are tested.

Videos

Insertion runs under different tablecloths and lighting, including a hand moving the rack.

Figures

From Figure 1: S2C splits contact-rich manipulation into two stages. In semantic localization, a VLM predicts a pre-c...
From Figure 1: S2C splits contact-rich manipulation into two stages. In semantic localization, a VLM predicts a pre-contact goal and motion planning moves the robot near the contact region. In contact refinement, human-in-the-loop real-world residual RL learns local force-aware corrections from wrist images with randomized backgrounds.
From Figure 2: The original scene, test-tube insertion on a black tablecloth.
From Figure 2: The original scene, test-tube insertion on a black tablecloth.
From Figure 2: The same task under one of the test conditions, with a patterned tablecloth and colored light.
From Figure 2: The same task under one of the test conditions, with a patterned tablecloth and colored light.
Abstract

Contact-rich manipulation requires precise control under complex contact dynamics while remaining robust to diverse visual conditions. However, real-world reinforcement learning often overfits to the visual appearance of training scenes. We propose Semantics-to-Contact (S2C), a stagewise framework for visually robust contact-rich manipulation. The insight is to decompose the problem into two stages: semantic focusing and contact refinement. In the first stage, a vision-language model localizes task-relevant regions across visually diverse scenes, providing coarse spatial guidance and reducing the burden of exploration. In the second stage, a real-world human-in-the-loop residual reinforcement learning policy learns fine-grained contact behaviors within the localized region using force feedback. To further improve visual robustness, we introduce an object-centric visual augmentation strategy that randomizes background appearance while preserving the robot and task-relevant objects. Experiments on six contact-rich real-world manipulation tasks exhibit that S2C consistently improves success rates over strong baselines and maintains reliable performance under significant visual and positional variations. These results demonstrate that stagewise semantic focusing and contact refinement provide a practical path toward visually robust real-world contact-rich manipulation. Our demos can be seen in https://edem-ai.github.io/S2C/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.