RL and sim-to-real
Reinforcement learning in simulation and transfer to real robots.
Start here: seminal papers
End-to-End Training of Deep Visuomotor Policies★ Must read
April 2015
Train a 92,000-parameter CNN that maps raw camera images straight to PR2 motor torques, using guided policy search to turn policy learning into supervised learning. End-to-end training beats policies built on separately trained vision on all four real tasks, e.g. 88.9% vs 55.6% success at screwing on a bottle cap.
Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw ImagesSeminal
June 2015
E2C trains a variational autoencoder whose latent dynamics are forced to be locally linear, so iLQR or AICO can plan directly on embeddings of raw images. It swings up a pendulum from 48x48 pixels in 90% of trials and balances a cart-pole and controls a three-link arm from images in 93% and 90%, where autoencoder baselines never succeed.
Continuous control with deep reinforcement learningSeminal
September 2015
DDPG adds DQN's replay buffer and target networks, plus batch normalization, to a deterministic actor-critic so it can learn continuous control. With one set of hyperparameters it solves more than 20 simulated physics tasks, often from raw pixels, nearly all within 2.5 million steps, and sometimes beats an iLQG planner that has the true dynamics.
Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy UpdatesSeminal
October 2016
Asynchronous NAF trains one deep Q-function from a replay buffer that several robots fill in parallel. Two 7-DoF arms learn to pull open a door from scratch, without demonstrations, in about 2.5 hours, and the final policy succeeds in 20 of 20 trials.
Deep Visual Foresight for Planning Robot MotionSeminal
October 2016
Train an action-conditioned video prediction model on 50,000 unlabeled robot pushes, then push objects by choosing actions whose predicted pixel motion carries a user-chosen pixel to its goal. On objects not seen in training it ends 2.52 pixels from the goal on average, against 3.19 for the best calibrated servoing baseline.
CAD2RL: Real Single-Image Flight without a Single Real ImageSeminal
November 2016
Train a quadrotor's collision-avoidance policy from single RGB images entirely in CAD hallways with randomized textures and lighting, then fly it in real buildings. In a realistic scanned hallway it flies 1.2 km without a collision in about 40% of runs, and in real flights it collides 6 times over 164 m versus 13 for a free-space-prediction baseline.
Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World★ Must read
March 2017
Train an object localizer only on simulated images with random non-realistic textures, lighting, camera poses and distractors. In the real world it localizes objects to about 1.5 cm and lets a Fetch robot pick targets out of clutter in 38 of 40 trials.
Safe Model-based Reinforcement Learning with Stability GuaranteesSeminal
May 2017
Model-based RL that improves a neural network policy using a Gaussian-process dynamics model and only collects data inside a region of attraction certified by a Lyapunov function. On a simulated inverted pendulum, the certified safe region grows from a small initial set over 50 data points and the pendulum never falls.
Deep reinforcement learning from human preferences★ Must read
June 2017
A human picks the better of two short clips of the agent, a neural reward model is fit to those choices, and RL optimizes that learned reward. With 700 comparisons it nearly matches RL on the true reward across eight MuJoCo tasks, and it learns a Hopper backflip from 900 comparisons in under an hour.
Hindsight Experience ReplaySeminal
July 2017
Replay each failed episode as if its goal had been the state the agent actually reached, so sparse binary rewards still give a learning signal. On a simulated Fetch arm, DDPG with HER solves pushing, sliding and pick-and-place where plain DDPG solves none, and the pick-and-place policy succeeds 5/5 on the real robot.
Emergence of Locomotion Behaviours in Rich EnvironmentsSeminal
July 2017
They train simulated walker, quadruped and humanoid bodies with a distributed PPO on procedurally generated obstacle courses, rewarding mostly forward velocity. Running, jumping, crouching and turning emerge without behaviour-specific reward shaping, and the Planar Walker learns to clear hurdles nearly as tall as its own body.
Proximal Policy Optimization Algorithms★ Must read
July 2017
PPO replaces TRPO's constrained update with a clipped probability-ratio objective that can be optimized for several epochs of minibatch SGD on the same batch. With clipping at 0.2 it scores 0.82 (normalized) on 7 MuJoCo tasks, vs at most 0.74 for KL-penalty variants and -0.39 with no clipping.
Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic GraspingSeminal
September 2017
Train a monocular-RGB grasp-success CNN on 8 million simulated grasps of random procedural shapes plus real data, with GraspGAN making the rendered images look real and a domain-adversarial loss aligning features. With 2% of the real data it matches the all-real model (68.51% vs 67.65%), and with no real labels it reaches 63.40%.
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations★ Must read
September 2017
Model-free RL learns four dexterous tasks on a simulated 24-DoF hand, and DAPG adds 25 VR demonstrations per task to policy gradient. With only a sparse success reward it reaches 90% success in 3-6 robot hours, against 16-98 hours for RL from scratch with shaped rewards.
Overcoming Exploration in Reinforcement Learning with DemonstrationsSeminal
September 2017
Add 100 VR demonstrations to DDPG with Hindsight Experience Replay through a critic-filtered behavior-cloning loss and resets to demonstration states. A simulated Fetch arm learns to stack 6 blocks (32% success), while behavior cloning and HER alone already get 0% on 4 blocks.
Sim-to-Real Transfer of Robotic Control with Dynamics RandomizationSeminal
October 2017
Train a recurrent puck-pushing policy in MuJoCo with 95 dynamics parameters randomized each episode, then run it on a real Fetch arm with no real-world training. The LSTM policy succeeds 0.89 of the time on the real robot versus 0.91 in simulation.
Asymmetric Actor Critic for Image-Based Robot LearningSeminal
October 2017
Train an image-based policy in simulation with a critic that sees the full simulator state while the actor sees only RGBD images. With domain randomization, the policy transfers to a real Fetch robot with no real-world training data and succeeds 5/5 on picking, pushing and block moving.
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor★ Must read
January 2018
Soft actor-critic (SAC) is an off-policy actor-critic that trains a stochastic policy to maximize reward plus entropy. On six continuous-control benchmarks it matches DDPG, PPO, SQL and TD3 on the easier tasks and beats them on the hardest, including the 21-action-dimension rllab Humanoid, where DDPG makes no progress.
Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for ResearchSeminal
February 2018
OpenAI adds 13 goal-conditioned, sparse-reward MuJoCo tasks to Gym for a Fetch arm and a Shadow Dexterous Hand, and benchmarks DDPG with and without Hindsight Experience Replay. DDPG+HER with sparse rewards does best, while plain DDPG often learns nothing on the hand tasks.
Addressing Function Approximation Error in Actor-Critic MethodsSeminal
February 2018
TD3 curbs value overestimation in DDPG with three changes, a target that takes the minimum of two critics, less frequent actor updates and noise on the target action. Across seven MuJoCo tasks it matches or beats DDPG, PPO, TRPO, ACKTR and SAC, and on Ant it reaches a return of 4372 where no baseline exceeds 1822.
Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement LearningSeminal
March 2018
VPG trains two fully convolutional Q-networks, one for pushing and one for grasping, by trial and error, so pushes are learned for how much they help later grasps. On challenging real-world block arrangements it reaches 83.3% grasp success versus 43.5% for a grasping-only policy, after about 5.5 hours of robot training.
World ModelsSeminal
March 2018
A VAE compresses each frame, an MDN-RNN predicts the next latent, and a linear controller with under 1,100 parameters is evolved on top. It scores 906 ± 21 on CarRacing-v0, and a VizDoom policy trained only inside the learned model survives 1092 ± 556 steps in the real game (750 counts as solved).
DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills★ Must read
April 2018
DeepMimic trains physically simulated characters with RL to imitate motion clips, rewarding closeness to the reference pose plus an optional task goal. One recipe learns 24 humanoid skills, from walking to backflips, and adding a task reward raises target-strike success from 19% (imitation only) to 99%.
Sim-to-Real: Learning Agile Locomotion For Quadruped RobotsSeminal
April 2018
Trains Minitaur quadruped gaits with PPO in PyBullet and narrows the sim-to-real gap with a measured actuator model, simulated latency, randomization and a compact observation space. The learned gallop and trot run on the real robot without fine-tuning, as fast as expert-tuned gaits with 35% and 23% less power.
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics ModelsSeminal
May 2018
PETS models the dynamics with an ensemble of probabilistic neural networks and plans through it with sampled particle trajectories and MPC. It reaches model-free asymptotic performance on four MuJoCo tasks in under 100 trials, with 8x fewer samples than SAC and 125x fewer than PPO on half-cheetah.
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation★ Must read
June 2018
Off-policy Q-learning on 580k real grasp attempts from 7 robots, choosing actions by optimizing the Q-function instead of training an actor. The closed-loop, RGB-only policy grasps 96% of unseen objects, against 78% for a prior self-supervised grasping method.
Learning Dexterous In-Hand Manipulation★ Must read
August 2018
An LSTM policy trained with RL entirely in heavily randomized simulation reorients a block in a real Shadow Dexterous Hand, sensing the block with motion capture or three cameras. It reaches a median of 13 consecutive goal orientations on the real hand (11.5 with vision), versus 0 for the same policy trained without randomization.
Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World ExperienceSeminal
October 2018
SimOpt alternates RL training in a GPU simulator with updates to the distribution of simulation parameters, so that simulated roll-outs of the policy match a few real-robot roll-outs. After two updates with 3 real roll-outs each, a Yumi swings a peg on a rope into a hole in 90% of 20 trials; after one update, a Panda opens a drawer in all 20 trials.
Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-CostSeminal
October 2018
Train neural-network policies for low-cost 3- and 4-fingered hands with model-free RL directly on the real robot, with no simulator. Valve turning, box flipping and door opening are learned in 7.4, 4.1 and 15.6 hours, and 20 human demonstrations cut valve and box training to 3.0 and 1.5 hours.
Learning Latent Dynamics for Planning from PixelsSeminal
November 2018
PlaNet learns a latent dynamics model from 64x64 images and picks every action by planning in that latent space, with no policy or value network. On six DeepMind control suite tasks it comes close to D4PG's final scores, and beats D4PG by 26% on cheetah, while using 1,000 episodes instead of 100,000.
Residual Reinforcement Learning for Robot ControlSeminal
December 2018
Add an RL policy on top of a hand-engineered feedback controller and train only that residual, directly on a real Sawyer arm. When the blocks in an insertion task are randomly tilted, residual RL succeeds in 15/20 trials versus 2/20 for the hand-engineered controller, after about three hours of real-world training.
Learning to Walk via Deep Reinforcement LearningSeminal
December 2018
Soft actor-critic with its temperature tuned automatically toward a target entropy, used to teach a Minitaur quadruped to walk from scratch on the real robot. A gait emerges in about two hours (about 400 rollouts) and handles a slope, wooden blocks and steps it never trained on.
Learning agile and dynamic motor skills for legged robots★ Must read
January 2019
Train ANYmal's controllers with RL in a simulator that uses a neural network, fitted to real robot data, to model the robot's series-elastic actuators, then deploy the policies directly on the robot. The learned locomotion policy tracks commands more accurately and with less torque than the best model-based controller, reaches 1.5 m/s (25% above the previous record), and gets the robot up after a fall.
Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich TasksSeminal
June 2019
An RL policy outputs both end-effector pose targets and the stiffness of an impedance controller (VICES), compared against joint-space and fixed-impedance action spaces on path following, door opening and wiping. VICES earns the highest wiping reward at a 28 N mean force, transfers from a simulated Panda to a Sawyer, and wipes a real whiteboard in 8 of 10 trials.
Deep Dynamics Models for Learning Dexterous ManipulationSeminal
September 2019
PDDM learns an ensemble of neural-network dynamics models and plans through them with a smoothed, reward-weighted MPC optimizer. On a real 24-DoF Shadow Hand it learns to rotate two Baoding balls from 4 hours of purely real-world data, with about 100% success on 90° turns.
RLBench: The Robot Learning Benchmark & Learning EnvironmentSeminal
September 2019
RLBench is a simulated benchmark of 100 hand-designed manipulation tasks for a Franka Panda, each with unlimited motion-planned demonstrations and text descriptions. The paper proposes a few-shot challenge on 10 held-out tasks but reports no baseline results.
Solving Rubik's Cube with a Robot HandSeminal
October 2019
A Shadow Hand policy and a vision state estimator are trained only in simulation with automatic domain randomization (ADR), which keeps widening the randomization ranges as the model improves. On the real hand the best policy completes a 43-move fair scramble in 20% of trials and half of it in 60%.
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement LearningSeminal
October 2019
Meta-World is an open-source MuJoCo benchmark of 50 distinct Sawyer manipulation tasks with randomized object and goal positions, built to test multi-task and meta-RL on broad task distributions. Seven existing algorithms struggle on it; the best multi-task method reaches 68.3% success on 10 tasks and 38.5% on 50, and no meta-RL method solves more than 39.9% of held-out tasks.
Dream to Control: Learning Behaviors by Latent ImaginationSeminal
December 2019
Dreamer learns a latent world model from pixels and trains an actor and a value model purely on imagined rollouts, backpropagating value gradients through the learned dynamics. On 20 DeepMind Control tasks it averages 823 after 5 million steps, above D4PG's 786 after 100 million.
robosuite: A Modular Simulation Framework and Benchmark for Robot LearningSeminal
September 2020
A modular MuJoCo framework that builds robot-learning tasks from interchangeable robots, grippers, controllers and objects, with nine standardized manipulation benchmarks. Soft Actor-Critic baselines solve three of the nine tasks and learn faster with operational-space control than with joint-velocity control.
Generation of GelSight Tactile Images for Sim2Real LearningSeminal
January 2021
They simulate a GelSight optical tactile sensor in Gazebo by turning a depth-camera image into a smoothed elastomer heightmap and shading it with Phong's model. A classifier trained only on generated images, with texture augmentation, recognises 21 real objects with 76.19% test accuracy.
Sim-to-Real for Robotic Tactile Sensing via Physics-Based Simulation and Learned Latent ProjectionsSeminal
March 2021
A finite-element model of the BioTac fingertip in GPU-based Isaac Gym matches a validated ANSYS model at 75x the speed. Small networks that project between learned latent spaces of simulated skin deformation and real electrode signals then predict real BioTac output for unseen objects and recover contact patches from real signals.
Tactile-RL for Insertion: Generalization to Objects of Unknown GeometrySeminal
April 2021
An RL policy learns on a real robot to correct peg-in-hole misalignment from two GelSlim tactile sensors, using a curriculum of increasingly constrained environments and marker-flow input. Trained on 4 objects, it inserts 4 unseen everyday objects with over 85% average success.
AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control★ Must read
April 2021
AMP replaces hand-designed motion-tracking rewards with a discriminator trained on unstructured mocap clips, whose score is added as a style reward to a simple task reward in RL. With a 56-clip locomotion dataset, a simulated humanoid picks walking, jogging or running gaits on its own and reaches 0.90 normalized return on target heading, versus 0.46 and 0.63 with walk-only or run-only data.
Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement LearningSeminal
May 2021
An LSTM policy trained with PPO in simulation on randomized staircases lets the Cassie biped climb and descend real stairs blind, using only proprioception and the existing flat-ground reward. On an outdoor staircase it succeeded in 8 of 10 ascents and 10 of 10 descents.
Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image TranslationSeminal
June 2021
Train tactile RL policies in PyBullet on simulated depth images of a TacTip sensor, then run them on a real robot by translating real tactile images into simulated ones with a pix2pix GAN. Zero-shot transfer works on 4 of 5 tasks, including 100 consecutive successful ball-rolling trials.
Habitat 2.0: Training Home Assistants to Rearrange their HabitatSeminal
June 2021
Habitat 2.0 is a physics-enabled home simulator with an interactive apartment dataset (ReplicaCAD) and a rearrangement benchmark for a Fetch robot, running at 25,734 steps per second on one 8-GPU node (850x real-time). On its tasks, flat RL fails at long-horizon rearrangement, a hierarchy of RL skills does better, and sense-plan-act is more brittle.
RMA: Rapid Motor Adaptation for Legged Robots★ Must read
July 2021
RMA trains a quadruped walking policy and an online adaptation module entirely in simulation; the module infers a latent environment code from the last 0.5 s of joint states and actions, so a Unitree A1 adapts in fractions of a second with no real-world fine-tuning. It carries 12 kg, its own body weight, and reaches 73.5% success in simulation versus 62.4% for domain randomization.
Isaac Gym: High Performance GPU-Based Physics Simulation For Robot LearningSeminal
August 2021
Isaac Gym keeps the whole RL loop on one GPU, with physics, observations, rewards and PPO training all on the device, and simulator state exposed as PyTorch tensors that never pass through the CPU. On a single A100 it trains Shadow Hand cube reorientation to 20 consecutive successes in under 1 hour; OpenAI needed 30 hours on 6144 CPU cores plus 8 V100 GPUs.
Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning★ Must read
September 2021
Run simulation and PPO training for thousands of quadrupeds in parallel on a single GPU, with a game-inspired terrain curriculum. A rough-terrain ANYmal policy trains in under 20 minutes (flat terrain in under 4) and transfers to the real robot.
A System for General In-Hand Object Re-Orientation★ Must read
November 2021
Model-free RL with teacher-student distillation trains a simulated Shadow Hand to reorient objects to any 3D orientation, palm up or palm down, without knowing their shape. On more than 2000 EGAD objects it succeeds 96% of the time palm up and over 80% palm down in mid-air.
Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged LocomotionSeminal
February 2022
Train a quadruped's locomotion policy and a neural state estimator together in one PPO loop, with the estimator supplying base velocity, foot height and contact probability to the policy. On a real Mini Cheetah it runs at up to 3.75 m/s on flat ground and 3.54 m/s on a slippery plate with friction 0.22.
Rapid Locomotion via Reinforcement LearningSeminal
May 2022
One RL policy trained in simulation, with an adaptive curriculum over velocity commands and teacher-student online system identification, runs the MIT Mini Cheetah at up to 3.9 m/s and spins it at 5.7 rad/s, deployed zero-shot.
MyoSuite -- A contact-rich simulation suite for musculoskeletal motor controlSeminal
May 2022
MyoSuite converts validated OpenSim models of the finger, elbow and hand into MuJoCo and builds 204 contact-rich RL tasks on them, from posing to key turning, pen twirling and Baoding balls, with sarcopenia, fatigue, tendon-transfer and exoskeleton variants. The converted models match OpenSim muscle forces within 4.1% RMS error while simulating 60x to 4000x faster.
Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement LearningSeminal
June 2022
Bi-DexHands is an Isaac Gym benchmark of 20 two-Shadow-Hand tasks matched to the ages at which children learn them, running at 30,000+ FPS on one RTX 3090. PPO solves most single tasks, but multi-task PPO averages 8.9 reward over all 20 tasks against 32.5 for per-task PPO, and meta RL barely beats a random policy.
DayDreamer: World Models for Physical Robot LearningSeminal
June 2022
Runs the Dreamer world-model RL algorithm online on four real robots, with no simulator and one set of hyperparameters. A Unitree A1 learns to roll over, stand up and walk in about 1 hour, and two arms learn visual pick-and-place from sparse rewards at close to human pick rates.
In-Hand Object Rotation via Rapid Motor Adaptation★ Must read
October 2022
Train a fingertip-only object-rotation policy in simulation on cylinders, plus a module that infers object properties from 1.5 s of joint history. Deployed without fine-tuning on an Allegro hand, it rotates over 30 real objects of 4.5-7.5 cm and 5-200 g, averaging 23.96 rad per 30 s on heavy objects.
DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to RealitySeminal
October 2022
Train an in-hand cube reorientation policy for a low-cost Allegro Hand entirely in Isaac Gym with automatic domain randomisation, and feed it cube poses from a vision model trained only on synthetic images. Over 10 real trials the best policy averages 27.8 consecutive successes (best run 112), using three RGB cameras and no markers.
Visual Dexterity: In-Hand Reorientation of Novel and Complex Object ShapesSeminal
November 2022
One RL policy trained in simulation uses a single depth camera to reorient new, complex objects to any orientation with a downward-facing, low-cost open-source hand, median time under 7 s. On a table, 55% of trials on unseen shapes end within 0.4 rad; in the air, an unseen duck is dropped 56% of the time.
Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of BehaviorSeminal
December 2022
Train one quadruped policy on flat ground only that can walk in many styles (gait, step frequency, body height and pitch, stance width, footswing height), so a human operator can retune it on the spot for stairs, curbs, a 22 cm bar or shoves instead of retraining. On unseen simulated platform terrain, trotting survives 0.88 of the episode vs 0.83 for a gait-free baseline.
Orbit: A Unified Simulation Framework for Interactive Robot Learning EnvironmentsSeminal
January 2023
An open-source framework on NVIDIA Isaac Sim for building robot learning environments with GPU-parallel rigid and deformable physics and photo-realistic rendering. It ships with 16 robots, 10 motion generators and over 20 tasks, and simulates about 10x faster than CPU-based frameworks on rigid-body tasks and 3x faster on cloth.
Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement LearningSeminal
April 2023
A 20-joint Robotis OP3 miniature humanoid learns 1v1 soccer with deep RL in simulation and plays on real robots zero-shot. Compared with the robot's scripted controllers it walks 181% faster, turns 302% faster, gets up in 63% less time and kicks 34% faster.
Eureka: Human-Level Reward Design via Coding Large Language ModelsSeminal
October 2023
GPT-4 writes RL reward functions as code from the environment's source, then rewrites them over several rounds using training statistics as feedback. Across 29 IsaacGym tasks the rewards beat expert human rewards on 83% of tasks, and with a curriculum they teach a simulated Shadow Hand to spin a pen.
Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion ControlSeminal
January 2024
One RL recipe trains walking, running and jumping controllers for the Cassie biped, using a policy that reads both a 2-second and a 4-step history of the robot's inputs and outputs, trained across many commanded tasks. Transferred zero-shot, Cassie ran 400 m in 2 min 34 s, 100 m in 27.06 s, and jumped 1.4 m forward and onto a 0.44 m platform.
Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust ManipulationSeminal
March 2024
RialTo scans a real scene into an articulated simulation, carries about 15 real demos into it, fine-tunes the imitation policy there with sparse-reward RL, and distills it back to a point-cloud policy. On real tasks it averages 91%, 77% and 75% success under pose randomization, distractors and disturbances, against 25%, 11% and 5% for imitation learning.
TacSL: A Library for Visuotactile Sensor Simulation and LearningSeminal
August 2024
TacSL simulates GelSight tactile images and contact force fields on the GPU inside Isaac Simulator, over 200x faster than prior tactile simulators, and adds a policy-learning toolkit with a new RL method (AACD). Tactile policies trained in simulation transfer zero-shot to a real robot, succeeding 91.4% of the time on peg placement and 82.7% on peg insertion.
SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian SplattingSeminal
September 2024
SplatSim swaps a simulator's mesh renderer for a Gaussian Splat of the real scene, moving each robot link and object's Gaussians to follow the PyBullet state. RGB diffusion policies trained only on these renderings transfer zero-shot to a real UR5 with 86.25% average success over four tasks, against 97.5% for policies trained on real data.
Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement LearningSeminal
October 2024
HIL-SERL trains vision-based manipulation policies with RL directly on real robots, starting from 20-30 demonstrations and taking corrections from a human with a SpaceMouse during training. It reaches 100% success on all 13 tasks, against 49.7% for an HG-DAgger baseline given the same human data, with 1.8x faster cycle times.
ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body SkillsSeminal
February 2025
ASAP learns a residual "delta action" model from real humanoid rollouts so the simulator reproduces real dynamics, then fine-tunes motion-tracking policies inside it. On a Unitree G1 this cuts global tracking error from 61.2 to 50.2 mm on a kick and from 159 to 112 mm on an unseen motion.
Recent
TactileLab: Sim-to-Real Dexterous Robot Manipulation with Efficient Shear-Sensitive Tactile Simulation
November 2026
TactileLab adds simulated tactile flow, a dense shear field computed from rigid-body contact motion in IsaacLab, so RL policies can feel tangential contact at scale. Trained only in simulation, a LEAP Hand policy using contact depth plus flow makes 1.95 real in-hand rotations in 30 s, against 0.82 with proprioception alone.
Semantics-to-Contact: A Stagewise Framework for Robust Contact-Rich Manipulation
November 2026
S2C uses a vision-language model to bring the robot near the contact point, then learns the final force-aware insertion with human-in-the-loop residual RL on the real robot, trained on wrist images with randomized backgrounds. Under new tablecloths and lighting it averages 69-84% success on six tasks, while HIL-SERL and HIL-ConRFT mostly fail.
Perceptive Humanoid Motion from Simulation
November 2026
HALO trains a Unitree G1 humanoid only on synthetic vision-language-motion data rendered in 1,000 procedurally generated rooms. Zero-shot, it sits, crouches, crawls, sidesteps through doors and pushes boxes in six unseen real homes, succeeding in 91% of 360 trials.
Reinforcement Learning Enables Scaling Generalist Robot Policy Improvement
November 2026
They post-train the pi0.5 VLA online with RL on up to 144 simulated pick tasks at once and test on held-out scenes and objects. With critic-based recipes the gains grow with the number of training tasks, up to +58 points on unseen scenes and +39 on unseen objects, while filtered fine-tuning generalizes much less.
Playing Soccer Like Humans Do: Learning Versatile and Agile Humanoid Soccer via Latent Motion Priors
November 2026
SoccerBot trains one skill-conditioned RL policy that moves, dribbles, traps and kicks a ball on a Unitree G1, acting as a residual on a latent motion prior distilled from human soccer motion capture. In simulation it kicks successfully 97.0% of the time and dribbles 89.9%, and it transfers zero-shot to the real robot with motion-capture ball tracking.
TwinBridge: Digital Twin-Driven Policy and Reward Transfer for Real-World Reinforcement Learning in Laboratory Robotics
November 2026
TwinBridge builds Gaussian-splatting digital twins of lab benches, pretrains a policy in them, and distills simulator-derived progress labels into a visual reward model for human-in-the-loop RL on the real robot. Across five lab tasks it reaches 98% mean autonomous success with at most 25 minutes of real-world interaction per task.
GSBR: Geometric Support Behavior Regularization for Online RL
November 2026
GSBR fine-tunes a frozen imitation-learned base policy with online RL, penalizing actions only by how far they fall outside the base policy's estimated support. Across 19 simulated tasks on four benchmarks it raises average success from 33% for the base policy to 95%, with about 23% higher normalized throughput than the next-best RL method.
DexHOT: Learning Hand-Object Configuration Transfer for Dexterous Manipulation
November 2026
One RL policy for a 22-DoF hand learns to move between grasps that fix both the finger joints and the object's pose in the hand, then chains these short moves into longer tasks. It succeeds on 61.5% of 200 real-robot transfers and 27 of 30 demonstration-guided task trials.
HLO-AMP: Hierarchical Loco-Manipulation from Object-Aware Motion Priors
November 2026
A Unitree G1 learns bimanual box pick-and-place from a low-level controller trained to imitate human motion capture with the box in the simulation, steered by a high-level RL policy. One deployed policy carries boxes up to 7 kg and recovers from drops. In simulation it succeeds 80.7% of the time, vs 50.3% for a whole-body controller trained without the object.
Play2Perfect: What Matters in Dexterous Play Pretraining for Precise Assembly?
June 2026
Pretrain one dexterous-hand RL policy by playing with a thousand random objects in simulation, then finetune it on sparse-reward assembly. Play2Perfect learns 33x faster than RL from scratch and transfers zero-shot to 0.5 mm insertions on a real robot.
New papers stay for 14 days.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.