Dexterous manipulation
Multi-fingered hands for grasping, in-hand reorientation and assembly.
Start here: seminal papers
Articulated Hands: Force Control and Kinematic IssuesSeminal
The International Journal of Robotics Research, March 1982
Kinematic and control design of the three-finger, 9-DOF Stanford-JPL hand. A mobility analysis picks the finger layout (39 of 600 candidate designs pass), isotropic points mark where fingertip forces are most accurate, and a Cartesian hand controller sets object stiffness and internal grasp forces without inverse kinematics.
Learning Dexterous Manipulation for a Soft Robotic Hand from Human DemonstrationSeminal
March 2016
Teaches the soft, pneumatic RBO Hand 2 from demonstrations that record only how a human moved the object. RL learns which demonstrations each controller can actually follow, and guided policy search merges the controllers into one neural network policy that turns a valve and pushes abacus beads from several hand positions, close to an oracle given the right demonstrations, and grasps a bottle in 10 of 10 trials.
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations★ Must read
September 2017
Model-free RL learns four dexterous tasks on a simulated 24-DoF hand, and DAPG adds 25 VR demonstrations per task to policy gradient. With only a sparse success reward it reaches 90% success in 3-6 robot hours, against 16-98 hours for RL from scratch with shaped rewards.
Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for ResearchSeminal
February 2018
OpenAI adds 13 goal-conditioned, sparse-reward MuJoCo tasks to Gym for a Fetch arm and a Shadow Dexterous Hand, and benchmarks DDPG with and without Hindsight Experience Replay. DDPG+HER with sparse rewards does best, while plain DDPG often learns nothing on the hand tasks.
Learning Dexterous In-Hand Manipulation★ Must read
August 2018
An LSTM policy trained with RL entirely in heavily randomized simulation reorients a block in a real Shadow Dexterous Hand, sensing the block with motion capture or three cameras. It reaches a median of 13 consecutive goal orientations on the real hand (11.5 with vision), versus 0 for the same policy trained without randomization.
Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-CostSeminal
October 2018
Train neural-network policies for low-cost 3- and 4-fingered hands with model-free RL directly on the real robot, with no simulator. Valve turning, box flipping and door opening are learned in 7.4, 4.1 and 15.6 hours, and 20 human demonstrations cut valve and box training to 3.0 and 1.5 hours.
Deep Dynamics Models for Learning Dexterous ManipulationSeminal
September 2019
PDDM learns an ensemble of neural-network dynamics models and plans through them with a smoothed, reward-weighted MPC optimizer. On a real 24-DoF Shadow Hand it learns to rotate two Baoding balls from 4 hours of purely real-world data, with about 100% success on 90° turns.
DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm SystemSeminal
October 2019
A glove-free teleoperation system that drives a KUKA arm and Allegro hand (23 actuated joints) from four depth cameras watching the operator's bare hand. Two pilots ran 15 dexterous tasks, such as pulling money out of a wallet, and 141 of 150 trials succeeded.
Solving Rubik's Cube with a Robot HandSeminal
October 2019
A Shadow Hand policy and a vision state estimator are trained only in simulation with automatic domain randomization (ADR), which keeps widening the randomization ranges as the model improves. On the real hand the best policy completes a 43-move fair scramble in 20% of trials and half of it in 60%.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation★ Must read
May 2020
The open-source, fingertip-sized GelSight-style sensor that made camera-based touch cheap and repeatable enough to put on multi-fingered hands. Shown driving learned tactile model-predictive control of glass marbles in hand.
DexYCB: A Benchmark for Capturing Hand Grasping of ObjectsSeminal
April 2021
A marker-less dataset of 10 people grasping 20 YCB objects: 582K RGB-D frames from 8 cameras, with 3D hand and object poses fit to crowdsourced keypoints. Hand-pose models trained on it transfer to HO-3D better than the reverse (1.4-1.9x versus 3.4-3.7x error increase), and it adds a benchmark for safe human-to-robot handover grasps.
DexMV: Imitation Learning for Dexterous Manipulation from Human VideosSeminal
August 2021
DexMV records human hand videos, estimates 3D hand and object poses, and retargets them into demonstrations for a 30-DoF simulated Adroit hand. With these demonstrations DAPG relocates four of five YCB objects at 93-100% success, where RL alone reaches 0-67%.
A System for General In-Hand Object Re-Orientation★ Must read
November 2021
Model-free RL with teacher-student distillation trains a simulated Shadow Hand to reorient objects to any 3D orientation, palm up or palm down, without knowing their shape. On more than 2000 EGAD objects it succeeds 96% of the time palm up and over 80% palm down in mid-air.
Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement LearningSeminal
June 2022
Bi-DexHands is an Isaac Gym benchmark of 20 two-Shadow-Hand tasks matched to the ages at which children learn them, running at 30,000+ FPS on one RTX 3090. PPO solves most single tasks, but multi-task PPO averages 8.9 reward over all 20 tasks against 32.5 for per-task PPO, and meta RL barely beats a random policy.
DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on SimulationSeminal
October 2022
A faster, more robust optimizer for force-closure grasps produces DexGraspNet, 1.32 million simulation-validated ShadowHand grasps on 5355 objects. Training the DDG grasp network on it instead of the earlier 6.9K-grasp dataset raises simulated success from 57.4% to 67.5%.
In-Hand Object Rotation via Rapid Motor Adaptation★ Must read
October 2022
Train a fingertip-only object-rotation policy in simulation on cylinders, plus a module that infers object properties from 1.5 s of joint history. Deployed without fine-tuning on an Allegro hand, it rotates over 30 real objects of 4.5-7.5 cm and 5-200 g, averaging 23.96 rad per 30 s on heavy objects.
DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to RealitySeminal
October 2022
Train an in-hand cube reorientation policy for a low-cost Allegro Hand entirely in Isaac Gym with automatic domain randomisation, and feed it cube poses from a vision model trained only on synthetic images. Over 10 real trials the best policy averages 27.8 consecutive successes (best run 112), using three RGB cameras and no markers.
Visual Dexterity: In-Hand Reorientation of Novel and Complex Object ShapesSeminal
November 2022
One RL policy trained in simulation uses a single depth camera to reorient new, complex objects to any orientation with a downward-facing, low-cost open-source hand, median time under 7 s. On a table, 55% of trials on unseen shapes end within 0.4 rad; in the air, an unseen duck is dropped 56% of the time.
VideoDex: Learning Dexterity from Internet VideosSeminal
December 2022
Retarget human hand motion from Epic Kitchens videos into robot hand and arm trajectories, pretrain an open-loop policy on them, then finetune on teleoperated demos. On a LEAP Hand and xArm, VideoDex picks unseen objects with 0.77 success versus at most 0.44 for baselines without the human-video prior.
UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned PolicySeminal
March 2023
UniDexGrasp samples diverse ShadowHand grasp poses from a tabletop point cloud, then executes one with a goal-conditioned RL policy distilled to point-cloud input. In simulation the full pipeline lifts 58-66% of thousands of objects, including unseen categories, while the best baseline policy reaches 25% even with oracle state.
Rotating without Seeing: Towards In-hand Dexterity through TouchSeminal
March 2023
Covers one side of an Allegro hand with 16 cheap binary touch sensors and trains an RL policy in simulation that rotates objects in the palm from touch and joint angles alone. On the real hand it turns a cuboid 4.91 rounds in 30 s versus 0.25 without touch, and rotates unseen objects such as an orange and a soup can.
AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation SystemSeminal
July 2023
One camera-based teleoperation system that drives different robot arms and dexterous hands, in simulation or the real world, needing only each robot's URDF. On the same xArm6 and Allegro hardware as a robot-specific prior system, it beats that system on 8 of its 10 real-world tasks and ties on the other 2.
LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot LearningSeminal
September 2023
A four-finger anthropomorphic robot hand built from Dynamixel motors, off-the-shelf brackets and 3D-printed parts in about 4 hours for 2000 USD, 1/8th the cost of Allegro. A new knuckle joint keeps sideways finger motion in every pose, and the hand matches or beats Allegro on 9 of 10 teleoperated tasks.
General In-Hand Object Rotation with Vision and TouchSeminal
September 2023
RotateIt trains an Allegro hand in simulation to rotate objects with its fingertips about the x, y or z axis, using a transformer that fuses segmented depth, fingertip contact and proprioception. Vision and touch lift x-axis rotation reward from 79 to 118 (privileged oracle 125), and the policy transfers to real objects without fine-tuning.
Eureka: Human-Level Reward Design via Coding Large Language ModelsSeminal
October 2023
GPT-4 writes RL reward functions as code from the environment's source, then rewrites them over several rounds using training statistics as feedback. Across 29 IsaacGym tasks the rewards beat expert human rewards on 83% of tasks, and with a curriculum they teach a simulated Shadow Hand to spin a pen.
Robot Synesthesia: In-Hand Manipulation with Visuotactile SensingSeminal
December 2023
Turn each triggered touch sensor into points on its mesh and merge them with the depth-camera point cloud, so one PointNet policy sees vision and touch in the same 3D space. Trained in simulation and transferred zero-shot to an Allegro hand, it reaches 22.9 rounds of real double-ball rotation, against 18.8 when touch is fed in as a separate vector.
DexCap: Scalable and Portable Mocap Data Collection System for Dexterous ManipulationSeminal
March 2024
A wearable rig of mocap gloves, SLAM wrist cameras and a chest RGB-D camera records human hand motion with 3D point clouds, and DexIL retargets it to two LEAP hands and trains a point-cloud diffusion policy. With no robot data, policies reach 72% average success on three tasks, and collection is three times faster than teleoperation.
Learning Visuotactile Skills with Two Multifingered HandsSeminal
April 2024
HATO is a low-cost VR teleoperation system for two UR5e arms fitted with touch-sensing prosthetic hands, used to train diffusion policies on vision, touch and proprioception. The policies succeed 9-10/10 on handover, stacking and pouring and 5/10 on steak serving, which drops to 0/10 without touch.
Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation LearningSeminal
July 2024
A real-time system that teleoperates two arms with dexterous hands from Apple Vision Pro hand tracking, with built-in collision and singularity avoidance and $1.2 vibration motors for touch feedback. Policies trained on its demonstrations average a 22% higher success rate than policies trained on AnyTeleop+ data.
DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation LearningSeminal
October 2024
DexMimicGen extends MimicGen to two-armed and dexterous-hand robots, turning 60 teleoperated demos into 21K simulated demos across nine tasks. On a real GR-1 humanoid, a policy trained on 40 generated demos sorts cans with 90% success, against 0% from the 4 source demos alone.
DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous ManipulationSeminal
May 2025
DexUMI records dexterous-hand demonstrations with an exoskeleton worn on the human hand that matches the robot hand's fingertip kinematics, then paints the robot hand over the human hand in the wrist-camera video. Policies trained only on this data average 86% success on four real tasks across two robot hands, and collection is 3.2x faster than teleoperation.
Recent
Latent RL Priors for Data-Efficient Dexterous Manipulation
November 2026
Distill a large-scale simulated RL policy for a KUKA arm with a dexterous hand into a frozen flow-matching action decoder, then learn only a visual front end from a few demonstrations. In simulation it beats π0.5 and GR00T N1.7 fine-tuned on the full datasets using 10x fewer demos on door lifting and 5x fewer on bimanual assembly.
PACT: Pose-Robust Actuation Co-Design for Tendon-Driven Robotic Hands
November 2026
PACT jointly picks tendon routing, moment arms and capstan sizes for a tendon-driven finger, so that one design covers the required fingertip forces even at its weakest pose while keeping 5 Hz joint bandwidth. For a human-sized four-DoF finger, it raises worst-pose task-force coverage from 0.24 to 0.88 over a hand-designed baseline. On the built prototype, measured force falls about 24.5% short of prediction because of tendon friction.
RoboTryOn: Synthesizing Dexterous Robot Demonstrations from Human Videos via Virtual Try-On
November 2026
RoboTryOn turns egocentric human videos into robot demonstrations by generating a video of the robot making the same motion and reading its actions back with an inverse dynamics model. On two bimanual robots with dexterous hands, it lifts success on seven tasks that have no robot demonstrations from 2% to 60% on average.
DexHOT: Learning Hand-Object Configuration Transfer for Dexterous Manipulation
November 2026
One RL policy for a 22-DoF hand learns to move between grasps that fix both the finger joints and the object's pose in the hand, then chains these short moves into longer tasks. It succeeds on 61.5% of 200 real-robot transfers and 27 of 30 demonstration-guided task trials.
DexPort: A Hand-Held Skill Interface for Dexterous Hand Data Collection
October 2026
A hand-held rig that carries the robot's own 20-DoF hand and camera, so demonstrations need no retargeting. Simple joystick skills drive the fingers, and the commanded joint targets, not measured states, are logged as actions; that alone lifts policy success from 11% to 64%.
Play2Perfect: What Matters in Dexterous Play Pretraining for Precise Assembly?
June 2026
Pretrain one dexterous-hand RL policy by playing with a thousand random objects in simulation, then finetune it on sparse-reward assembly. Play2Perfect learns 33x faster than RL from scratch and transfers zero-shot to 0.5 mm insertions on a real robot.
New papers stay for 14 days.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.