SimDuoReal: Object-Centric Bimanual Policy for Emergent Dexterity

Qiyang Yan, Yulin Liu, Akash Sharma, Haozhi Qi, Guanya Shi, C. Karen Liu

November 2026

Real robot, barcode scanning: the hands pick up a brush, pass it across and turn its barcode toward the scanner.

TL;DR One bimanual policy for two arms with 22-DoF hands is trained in simulation with a single object-pose reward, and pickup, regrasping, in-hand adjustment and handover emerge from how goals are sampled. Prompted only with object motion, it succeeds in 61 of 80 real trials (76.3%) on four tasks with unseen objects.

What they do

Bimanual dexterous systems usually specify which hand acts and when the object changes hands, through task-specific rewards or demonstrations. SimDuoReal extends the object-centric interface of SimToolReal to two arms with multi-fingered hands: the policy is told only where each object should go (6-DoF pose goals), and the coordination comes from how training goals are sampled, not from extra reward terms.

How it works

Three experts, one reward. Left and right single-arm experts get goals inside their own workspace and learn pickup, reorientation and regrasping. A bimanual expert starts with one hand holding the object and gets a goal in the other hand’s workspace; the goal counts only once the receiver touches the target region and the giver releases, so handover is the only way to succeed.
Object-centric features. Displacements from hand keypoints to target and avoidance regions on the object, plus inter-hand displacements.
Distillation. Experts are refined on states collected at expert transitions, then distilled into one policy with DAgger plus an RL loss, over scenes covering handover, end-to-end transfer and concurrent two-object manipulation.
Deployment. Two Franka Research 3 arms with 22-DoF Sharpa Wave hands, three ZED cameras and FoundationPose tracking. Goals come from a VR operator moving virtual objects or from human video.

Results

Real robot, unseen objects, 20 trials per task: barcode scanning 15/20, peg insertion 15/20, cleaning 16/20, cooking 15/20, in total 61/80 (76.3%).
Simulated barcode scanning over 3,072 environments: 85.0% success, against 60.4% for chained experts without the handover expert.
Transition-aware refinement lifts the chained teachers from 79.4% to 84.3%. Dropping the RL loss gives 81.6%; dropping the handover expert scenes gives 68.3%.

Limitations

The system needs accurate object meshes and multi-view pose tracking. The 6-DoF goal interface does not capture force- or timing-dependent behaviors, and goal trajectories are supplied externally, not generated by the robot.

Videos

Real robot, cleaning: one hand holds a brush and sweeps toward a dustpan.

Simulation rollouts across parallel environments with primitive objects.

Figures

From Figure 1: System overview. (A) Three experts are trained in simulation under a shared object-centric representat... — From Figure 1: System overview. (A) Three experts are trained in simulation under a shared object-centric representation and reward, with different goal-sampling regimes: the left and right experts learn grasping with in-hand manipulation from goals inside one arm’s workspace, and the handover expert learns handover from goals that shift from the giver’s workspace to the receiver’s. (B) The experts are refined from a pool of states collected at expert transitions, then distilled into a unified policy with DAgger and RL. (C) At deployment, object trajectories from sources such as VR teleoperation and videos serve as the interface to the single policy.

From Figure 2: The object-centric representation during handover. The object’s surface splits into the target contact... — From Figure 2: The object-centric representation during handover. The object’s surface splits into the target contact region (handle, green) and the non-target contact region (head, red); nearest-region displacements relate the hand keypoints of the target actor (receiver, blue) and the non-target actor (giver, orange) to both regions.

From Figure 4: Emergent behaviors on real hardware. (a) Handover: the giver holds a stable grasp, adjusts the brush i... — From Figure 4: Emergent behaviors on real hardware. (a) Handover: the giver holds a stable grasp, adjusts the brush in-hand to expose a graspable region for the receiver, and releases once the receiver has grasped and settled into a more stable grasp. (b) Pre-grasp manipulation: when the object rests in a pose that is not graspable, the hand first pushes it into a graspable pose. (c) In-hand manipulation: the hand reorients the brush within the grasp to track subsequent object goals. None of these behaviors is specified explicitly; the policy discovers them from object-pose goals alone.

From Figure 5: Expert training vs. end-to-end training. Episode reward over training frames for the cross-workspace e... — From Figure 5: Expert training vs. end-to-end training. Episode reward over training frames for the cross-workspace expert and for a single policy trained end-to-end.

Abstract

Bimanual dexterous manipulation requires coordinated arm and hand control, yet specifying such coordination manually is challenging and often task-specific. In this work, we explore whether complex bimanual dexterity can emerge from a simple object-centric learning objective. We train reinforcement learning policies under a shared object-centric reward, varying their training environments rather than tuning separate rewards for each behavior. We find that the policies discover pre-grasp manipulation, grasping and regrasping, in-hand adjustment and handover to reach object-pose goals without hand-designed motion sequences. We then refine the experts on transition states and use DAgger and reinforcement learning to distill them into a single policy that composes these behaviors for diverse bimanual tasks. On real hardware, the same policy succeeds in 61 of 80 trials (76.3\%) across four tasks with unseen objects and no task-specific fine-tuning. In simulated barcode scanning, our policy achieves 85.0\% success, compared with 60.4\% for two single-arm policies without handover. These results suggest that this objective provides a simple and scalable interface for learning general bimanual dexterity, shifting the burden from manually designing robot coordination to specifying what should happen to the object.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong