SyncHeC: Markerless Spatio-Temporal Hand-Eye Calibration via Differentiable Rendering and Continuous-Time Joint Optimization

Zhengdong Hong

November 2026

TL;DR SyncHeC finds where a camera sits relative to a robot arm and how late its images are relative to the joint log, from one 6.5 s motion and no markers, by matching rendered robot masks and depth to the images. It reaches 0.21 cm and 2.7 ms error, and correcting the delay raises real policy success by 8.1 points over 1,800 paired trials.

What they do

Using camera images for control requires knowing where the camera is relative to the robot (the hand-eye extrinsics). It also requires knowing which joint state each image shows: exposure, transport and clock offsets shift each image’s timestamp, by 66 to 115 ms on the sensors tested. SyncHeC estimates both from one short continuous robot motion, without markers, for external (Eye-to-Hand) and wrist (Eye-in-Hand) cameras, and for several cameras at once.

How it works

Spatial anchor. From one static RGB-D frame, SAM2 segments the robot base, candidate camera poses are sampled on a sphere sized from depth, and the best one is refined by rendering the robot mask.
Differentiable delay. The 250 Hz joint log is interpolated in time, so gradients from rendered masks and depth reach the unknown offset through forward kinematics. Sensor depth, with holes filled by scaled Depth Anything V2 predictions, constrains the pose along the camera axis.
Planned motion. RRT* and a B-spline optimizer produce a 6.5 s trajectory that gives both varied views and enough image motion to tell a time shift from a pose change.
Alternating updates. Adam updates the delay and the pose in turn, converging in 3-5 cycles.

Results

On an xArm7 (five recordings), Eye-to-Hand errors are 0.21 cm, 0.15° and 2.7 ms against an 87 ms reference; CM-Calib on the same data gives 0.78 cm and 6.2 ms. EasyHeC++ applied to the moving robot without delay correction gives 3.92 cm.
Across seven arms and four depth sensors, estimated delays are within 3 ms of their references. The whole pipeline takes about eight minutes.
Six policy families (ACT, DP, π0, DP3, ManiFlow and a world-model MPC) on six real tasks: correcting the delay at training and deployment raises successes from 681 to 827 of 1,800 matched pairs (+8.1 points, 95% CI 6.3-9.9), and every policy-task cell improves.

Limitations

It needs a starting pose within roughly 15 cm, a visible robot and enough motion; blur becomes a problem above about 0.5 m/s. The delay is assumed constant, so bus, driver or host changes that shift it by more than 20 ms call for recalibration. The policy study uses one training seed per policy-task cell.

Figures

From Figure 1: Hand-eye calibration has two coupled parts. Spatial calibration recovers the camera pose in the robot ... — From Figure 1: Hand-eye calibration has two coupled parts. Spatial calibration recovers the camera pose in the robot frame, for Eye-to-Hand and Eye-in-Hand cameras; temporal calibration recovers the delay between when a frame is stamped and the joint state it actually shows. SyncHeC solves both jointly.

From Figure 2: Overview. Stage 1 loads the robot URDF into a simulator, samples candidate camera poses around it usin... — From Figure 2: Overview. Stage 1 loads the robot URDF into a simulator, samples candidate camera poses around it using the first depth frame, and refines the best one by differentiable rendering. Stage 2 runs one planned continuous trajectory, treats the high-rate joint log as continuous-time kinematics, and jointly optimizes the camera pose and the delay with a mask-and-depth rendering loss.

From Figure 4: Calibrating two cameras at once. Left: the rig with a tripod-mounted Eye-to-Hand camera and a wrist-mo... — From Figure 4: Calibrating two cameras at once. Left: the rig with a tripod-mounted Eye-to-Hand camera and a wrist-mounted Eye-in-Hand camera. Right: frames from each camera over one continuous trajectory.

Abstract

Markerless hand-eye calibration under continuous motion requires jointly resolving camera geometry and the time offset between images and joint states. We present SyncHeC (Synchronized Hand-Eye Calibration), a dense differentiable-rendering framework that estimates both from a single robot trajectory. Continuous-time forward kinematics makes latency differentiable, metric depth complements mask contours, and an offline planner combines spatial sensitivity with temporal excitation. Alternating updates support Eye-to-Hand, Eye-in-Hand, and multi-camera calibration. On a 7-DoF arm, one $6.5$\,s motion yields $0.21$\,cm translation error against a marker-board reference, $0.34$\,cm touch-test error, and $2.7$\,ms latency error relative to an $87$\,ms reference with $95\%$ uncertainty of $\pm1.5$\,ms. Against CM-Calib on matched recordings, translation and nominal latency errors improve by $3.7\times$ and $2.3\times$. Experiments span seven arms and four depth sensors. Compensating the recovered offset during policy training and deployment increases success by $8.1$ percentage points across six families and six tasks ($1{,}800$ matched pairs; paired $95\%$ confidence interval $[6.3,9.9]$ points).

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong