TCP4D: A Unified Tool-Center-Point Tracking and Generation Model for Robot Manipulation

TL;DR TCP4D recovers the 6-DoF tool-center-point pose and gripper state of a robot from monocular RGB video, and reuses the same 4D scene features to generate future tool trajectories as a policy. Tracking error is 7.60 mm on DROID, and on two real-robot tasks the policy succeeds 78.33% of the time, against 63.33% for DP3 and 31.67% for ACT.
What they do
Many robot videos lack action labels. TCP4D uses the tool-center-point (TCP) pose as an embodiment-agnostic action. One model tracks it in video, generates its future trajectory as a policy, and turns videos of one robot into demonstrations for another.
How it works
- Tracker. A DINOv2-based backbone initialized from 4RC encodes the video into a 4D scene representation, predicting metric depth and camera rays. From one 2D query per arm in the first frame, it decodes the TCP pose and gripper state at every frame, with no rigid fitting of point tracks.
- Generator. Features are sampled along the tracked TCPs and combined with robot state, wrist-camera and language features in a flow-matching DiT. Four Euler steps predict the next 32 TCP and gripper states, which are executed through IK.
- Augmentation. Tracked trajectories are retargeted to a target robot by IK. SAM3 (or the projected source URDF) masks the source robot, video inpainting fills the background, and the rendered target robot is composited in.
Results
- Tracking: ATE of 7.60 mm on DROID and 16.07 mm on unseen RoboLab videos, against 67.65 and 70.90 mm for the best baselines, which fit TCP poses to tracked points. Fine-tuning on ten UR5 episodes gives 8.85 mm ATE and 98.48% gripper accuracy.
- RoboTwin (48 tasks, clean training and test): 69.19% mean success, against 54.27% for DP3 and 45.21% for π0. With randomized test scenes it drops to 10.94%.
- Augmentation: joint training on source and augmented data raises mean success on ARX-X5 from 30.5% to 72.0% and on UR5-WSG from 32.0% to 39.0%, compared with augmented-only training.
- Real robot (CoBoT Magic, 60 demonstrations per task, 30 trials): 83.33% on placing pens and 73.33% on stacking bowls.
Limitations
The tracker’s pretraining data covers only Franka arms with Robotiq grippers, so other robots need few-shot fine-tuning. The generator is trained on small sets of clean demonstrations without large-scale pretraining and generalizes poorly to randomized scenes. The real-robot evaluation has only two tasks on one platform.
Figures




Abstract
Robot manipulation videos provide a rich visual experience, yet recovering robot action and gripper-state annotations from RGB observations still remains challenging. Using the embodiment-agnostic tool-center-point (TCP) pose as the action space, we present TCP4D, a unified model in which a shared 4D scene representation both tracks observed TCP motion and generates future TCP trajectories. Given a monocular video and initial 2D TCP queries, the tracker directly recovers metric TCP poses in SE(3) and gripper states alongside scene geometry and camera parameters. The generator samples local features from the same 4D representation along the recovered TCP tracks, then combines them with robot observations and language instructions to predict future TCP states. Our new cross-embodiment augmentation pipeline converts existing RGB videos into demonstrations for alternative embodiments by retargeting the recovered motion and synthesizing the corresponding observations, without recorded source-robot states. Experiments show that the tracker achieves absolute trajectory errors of 7.60 and 16.07 mm on DROID and RoboLab and adapts to a new embodiment with only ten demonstrations. The generator achieves 69.19% mean success on RoboTwin and outperforms well-established policy models on two real-world tasks, while joint training on source and augmented demonstrations improves target-robot success by 41.5% over augmented-only training.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
