HOC2R: Hand-Object Contact Trajectories for Human-to-Robot Motion Transfer

Hongkun Yu, Francisco M. López, Francisco Cruz

November 2026

TL;DR HOC2R finds where fingers touch objects in human manipulation videos, using small heads on frozen DINOv3 and SAM 2.1 features, and turns the contacts into gripper trajectories. On a real Kinova arm it succeeds on 3 of 6 sequences, against 1 of 6 for each adapted baseline.

What they do

Human videos show plenty of manipulation but not where a gripper should touch or how it should move. HOC2R extracts hand-object contact trajectories from such videos: the visible finger-object contact points in each frame, linked over time. The midpoint of two contacts is lifted to 3D and replayed by a two-finger gripper as post-contact motion.

How it works

Four light heads on frozen features. A DINOv3 head proposes the ten likeliest 32×32 contact regions. SAM 2.1 heads score each box for contact, count visible contacts (0, 1 or 2) and predict a whole-image contact heatmap. Touch scores rerank the heatmap peaks.
Two modes. Count-aware automation handles untrimmed video. Fixed-two localization assumes the contact frames are known and always returns two points inside proposal boxes.
To the robot. The contact midpoint is deprojected with RealSense depth into the Kinova base frame and filtered for bad depth, workspace violations and jumps. The arm then executes it with inverse kinematics.
Data. 210 point-annotated frames from 30 Human2Robot tasks.

Results

Held-out tasks: contact-count accuracy is 71.4% vs 58.1% for adapted VRB2, F1@10 is 62.0% vs 25.8%, and mean error is 10.46 vs 19.81 px.
Seven unseen RGB-D rollouts with natural-skin hands: the five-task fixed-two checkpoint reaches 27.02 px mean error, vs 67.67 px for VRB2.
Robot: the 30-task checkpoint gives 4 feasible trajectories and succeeds on 3 of 6 sequences (grasp-and-place, push, pen motion). VRB2 and EgoHOS-Point succeed on 1 each.

Limitations

The data is small: 210 annotated frames, one camera system and seven external rollouts. Only two-finger contacts are modeled, frame by frame. Count-aware mode breaks under domain shift (it misses 63 of 129 positive external frames), so the better mode needs contact frames given. The robot tests are six sequences that start from an established grasp and use open-loop inverse kinematics.

Figures

From Figure 3: On a human pen-motion demonstration, the midpoints of the two contacts HOC2R extracts form a 2D trajec... — From Figure 3: On a human pen-motion demonstration, the midpoints of the two contacts HOC2R extracts form a 2D trajectory.

From Figure 3: The trajectory is mapped into the robot base frame with aligned depth and camera calibration, filtered... — From Figure 3: The trajectory is mapped into the robot base frame with aligned depth and camera calibration, filtered, and executed on the arm with inverse kinematics.

From Figure 4: F1@10 and mean localization distance, with 95% confidence intervals, on seven external RGB-D rollouts ... — From Figure 4: F1@10 and mean localization distance, with 95% confidence intervals, on seven external RGB-D rollouts for VRB2, EgoHOS and three HOC2R checkpoints.

From Figure 5: In a two-hand pencil scene, HOC2R proposal boxes focus on the manipulating hand and the fixed-two deco... — From Figure 5: In a two-hand pencil scene, HOC2R proposal boxes focus on the manipulating hand and the fixed-two decoder localizes the pencil contact, with an error of 17.81 px vs 156.11 px for VRB2 and 126.00 px for EgoHOS-Point.

Abstract

Human videos contain abundant object-manipulation demonstrations, but they do not directly provide motion cues that a robot can follow. We present HOC2R, a Hand-Object Contact trajectory extractor for Robot motion that combines frozen DINOv3 and SAM 2.1 features with learned heads for hand-object association and contact localization. HOC2R supports two modes: count-aware automation that detects, counts, and localizes visible contacts in untrimmed videos, and fixed-two localization for two-finger grippers when contact frames are known. On a 30-task point-annotated benchmark, count-aware automation improves contact-count accuracy from 58.1% to 71.4% and reduces mean localization distance from 19.81 to 10.46 pixels relative to an adapted VRB2 baseline. On seven unseen RGB-D rollouts, a five-task fixed-two checkpoint reduces mean localization distance from 67.67 to 27.02 pixels. Across six robot evaluation sequences, the 30-task fixed-two checkpoint produces four feasible trajectories and succeeds on three tasks, compared with two feasible trajectories and one successful task for each adapted baseline. HOC2R is competitive in the evaluated single-hand scenes, robust to the hand-color shift, and strongest in the evaluated multi-hand scenes. These results support HOC2R as a practical pseudo-labeling method for extracting robot-usable contact trajectories from human demonstrations.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong