← Paper Digest

TCP4D: A Unified Tool-Center-Point Tracking and Generation Model for Robot Manipulation

Haosong Peng, Xiangyang Liu, Zhengshen Zhang, Yunfeng Fan, Tianyu Qi, Xin Xia, Yufeng Zhan, Wenchao Xu

November 2026

TL;DR TCP4D recovers the 6-DoF tool-center-point pose and gripper state of a robot from monocular RGB video, and reuses the same 4D scene features to generate future tool trajectories as a policy. Tracking error is 7.60 mm on DROID, and on two real-robot tasks the policy succeeds 78.33% of the time, against 63.33% for DP3 and 31.67% for ACT.

What they do

Many robot videos lack action labels. TCP4D uses the tool-center-point (TCP) pose as an embodiment-agnostic action. One model tracks it in video, generates its future trajectory as a policy, and turns videos of one robot into demonstrations for another.

How it works

  • Tracker. A DINOv2-based backbone initialized from 4RC encodes the video into a 4D scene representation, predicting metric depth and camera rays. From one 2D query per arm in the first frame, it decodes the TCP pose and gripper state at every frame, with no rigid fitting of point tracks.
  • Generator. Features are sampled along the tracked TCPs and combined with robot state, wrist-camera and language features in a flow-matching DiT. Four Euler steps predict the next 32 TCP and gripper states, which are executed through IK.
  • Augmentation. Tracked trajectories are retargeted to a target robot by IK. SAM3 (or the projected source URDF) masks the source robot, video inpainting fills the background, and the rendered target robot is composited in.

Results

  • Tracking: ATE of 7.60 mm on DROID and 16.07 mm on unseen RoboLab videos, against 67.65 and 70.90 mm for the best baselines, which fit TCP poses to tracked points. Fine-tuning on ten UR5 episodes gives 8.85 mm ATE and 98.48% gripper accuracy.
  • RoboTwin (48 tasks, clean training and test): 69.19% mean success, against 54.27% for DP3 and 45.21% for π0. With randomized test scenes it drops to 10.94%.
  • Augmentation: joint training on source and augmented data raises mean success on ARX-X5 from 30.5% to 72.0% and on UR5-WSG from 32.0% to 39.0%, compared with augmented-only training.
  • Real robot (CoBoT Magic, 60 demonstrations per task, 30 trials): 83.33% on placing pens and 73.33% on stacking bowls.

Limitations

The tracker’s pretraining data covers only Franka arms with Robotiq grippers, so other robots need few-shot fine-tuning. The generator is trained on small sets of clean demonstrations without large-scale pretraining and generalizes poorly to randomized scenes. The real-robot evaluation has only two tasks on one platform.

Figures

From Figure 1: TCP4D tracks tool-center-point (TCP) poses and gripper states from RGB videos and generates task-condi...
From Figure 1: TCP4D tracks tool-center-point (TCP) poses and gripper states from RGB videos and generates task-conditioned future TCP trajectories, used for robot manipulation and for cross-embodiment data augmentation.
From Figure 2: (1) The Tracker builds a 4D scene representation from a monocular video and, given the TCP's 2D locati...
From Figure 2: (1) The Tracker builds a 4D scene representation from a monocular video and, given the TCP's 2D location in the first frame, decodes scene geometry, the TCP's SE(3) motion and the gripper state. (2) The Generator samples features along the TCP tracks and, conditioned on the scene representation, wrist view and language instruction, predicts TCP motion and gripper state for the next steps.
From Figure 6: Task success rates of policies trained on source data only, augmented data only, and both, on three em...
From Figure 6: Task success rates of policies trained on source data only, augmented data only, and both, on three embodiments. Each task has 50 source demonstrations on Aloha-AgileX, augmented to ARX-X5 and UR5-WSG.
From Figure 7: Real-world setup and rollouts of placing pens and stacking bowls, with the predicted TCP trajectory ov...
From Figure 7: Real-world setup and rollouts of placing pens and stacking bowls, with the predicted TCP trajectory over the next 32 steps shown in each inference window (blue to orange in time).
Abstract

Robot manipulation videos provide a rich visual experience, yet recovering robot action and gripper-state annotations from RGB observations still remains challenging. Using the embodiment-agnostic tool-center-point (TCP) pose as the action space, we present TCP4D, a unified model in which a shared 4D scene representation both tracks observed TCP motion and generates future TCP trajectories. Given a monocular video and initial 2D TCP queries, the tracker directly recovers metric TCP poses in SE(3) and gripper states alongside scene geometry and camera parameters. The generator samples local features from the same 4D representation along the recovered TCP tracks, then combines them with robot observations and language instructions to predict future TCP states. Our new cross-embodiment augmentation pipeline converts existing RGB videos into demonstrations for alternative embodiments by retargeting the recovered motion and synthesizing the corresponding observations, without recorded source-robot states. Experiments show that the tracker achieves absolute trajectory errors of 7.60 and 16.07 mm on DROID and RoboLab and adapts to a new embodiment with only ten demonstrations. The generator achieves 69.19% mean success on RoboTwin and outperforms well-established policy models on two real-world tasks, while joint training on source and augmented demonstrations improves target-robot success by 41.5% over augmented-only training.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.