VOKE: Visual Object Kinematics Estimator for Dexterous In-Hand Manipulation

Shengjia Zhu, Wenqiang Xu, Lin Shao

November 2026

Live VOKE velocity and acceleration estimates during real-world in-hand cube reorientation.

TL;DR VOKE plugs into existing hand-object pose estimators and predicts an object's linear and angular velocity and acceleration from the last three frames, then uses them to refine the pose. It cuts linear-acceleration error on DexYCB from 6.18 to 1.31 m/s² compared with differentiating poses, and a dexterous-hand policy given its estimates succeeds 78.9% of the time in simulation, against 68.2% with pose history.

What they do

Pose estimators report where an object is, not how it moves, and differentiating noisy pose estimates amplifies the noise, especially for acceleration. VOKE attaches to an existing hand-object pose estimator and outputs the object’s linear and angular velocity and acceleration, then uses them to refine the pose. As an application, an in-hand cube-reorientation policy that observes this full kinematic state is trained in simulation and run on a physical robot.

How it works

PKM. Shares the base estimator’s image encoder, concatenates the features of the last k = 3 frames and regresses a 12-D motion vector with a small MLP. It is trained against velocities and accelerations derived from spline fits of the ground-truth pose trajectories.
KPRM. Integrates the previous pose with the predicted motion into a second-order reference pose. An MLP takes this reference, the motion vector and the recent pose estimates and outputs the refined pose.
KARL. A PPO policy for a Shadow Hand in Isaac Sim observes pose, velocity and acceleration instead of a three-frame pose history. It runs on the real robot without fine-tuning, with VOKE processing a live Azure Kinect RGB stream.

Results

Added to gSDF, HFL-Net and HOISDF on DexYCB and OakInk, VOKE improves every available pose metric. It cuts translational relative pose error by 41.0-64.4% and rotational by 7.9-33.3%.
It has lower velocity and acceleration error than Savitzky-Golay pose differencing for all four quantities on all three datasets. On DexYCB, linear-acceleration error drops from 6.18 to 1.31 m/s².
With VOKE-predicted state in simulation, the policy succeeds in 78.9% of episodes, against 68.2% with pose history. Final rotation error is 8.90° against 12.75°. With ground-truth state the two are close (98.1% vs 97.2%).
The real-robot deployment is shown qualitatively only.

Limitations

VOKE is limited by its backbone: severe occlusion or extreme viewpoints corrupt both the motion estimates and the refinement. Angular acceleration stays hard to estimate under rapid contacts and slipping. The authors have not yet measured how estimation errors affect real-world closed-loop manipulation.

Videos

VOKE estimates against reference values on DexYCB and OakInk sequences.

VOKE estimates next to the pose-differencing baseline on DexYCB and OakInk.

Figures

From Figure 1: VOKE adds linear and angular velocity and acceleration to frame-wise object pose estimation (top). In ... — From Figure 1: VOKE adds linear and angular velocity and acceleration to frame-wise object pose estimation (top). In real-world deployment it turns live robot RGB(D) frames into kinematics estimates for dexterous in-hand manipulation (bottom).

From Figure 2: A base pose estimator shares its image encoder with the Pluggable Kinematics Module (PKM), which decod... — From Figure 2: A base pose estimator shares its image encoder with the Pluggable Kinematics Module (PKM), which decodes linear and angular velocity and acceleration from the features of the last k frames. The refinement module (KPRM) builds a reference pose from the previous pose and these motion estimates, then refines the current pose. The refined pose and motion estimates feed the KARL policy along with robot proprioception.

From Figure 3: VOKE on DexYCB and OakInk. Left: the last three input frames. Right: linear and angular velocity and a... — From Figure 3: VOKE on DexYCB and OakInk. Left: the last three input frames. Right: linear and angular velocity and acceleration, with reference values from ground-truth poses in green, VOKE estimates in red and refined-pose boxes in light blue.

From Figure 4: Online VOKE predictions while the simulation-trained policy runs on the real robot. Left: three consec... — From Figure 4: Online VOKE predictions while the simulation-trained policy runs on the real robot. Left: three consecutive input frames. Right: predicted linear and angular motion, with velocity in blue and acceleration in red.

Abstract

Vision-based dexterous in-hand manipulation requires knowing not only where an object is, but also how it moves. However, most robotic manipulation systems rely on off-the-shelf visual perception models whose outputs are limited to static object masks, geometry, or 6D poses, leaving downstream controllers to infer object dynamics implicitly from noisy visual observations. We present VOKE, a kinematics-aware perception framework that augments existing object and hand-object pose estimators to recover a complete object kinematic state comprising pose, linear and angular velocity, and linear and angular acceleration. VOKE contains two core perception modules and a downstream application study. First, a Pluggable Kinematics Module (PKM) attaches to the output of a standard pose estimator and predicts the object's linear and angular velocity and acceleration from temporal visual features. Second, a Kinematics-guided Pose Refinement Module (KPRM) converts this motion vector into a second-order, motion-prior reference pose and refines the per-frame pose estimates. We evaluate VOKE on DexYCB and OakInk, where it consistently improves object pose estimation across three different backbones and substantially reduces velocity and acceleration estimation error compared with pose differencing. As a downstream application, we use Kinematics-Augmented Reinforcement Learning (KARL) to compare a kinematics-augmented policy with a pose-history baseline. After training the policies in simulation, we transfer the kinematics-augmented policy to the physical robot without policy fine-tuning, demonstrating the complete perception-to-control pathway in the target domain.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong