Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation

Marcel Torne, Anthony Simeonov, Zechu Li, April Chan, Tao Chen, Abhishek Gupta, Pulkit Agrawal

Seminal March 2024

Pipeline: 3D scene reconstruction, real-to-sim policy transfer, RL fine-tuning in sim, real-world deployment.

TL;DR RialTo scans a real scene into an articulated simulation, carries about 15 real demos into it, fine-tunes the imitation policy there with sparse-reward RL, and distills it back to a point-cloud policy. On real tasks it averages 91%, 77% and 75% success under pose randomization, distractors and disturbances, against 25%, 11% and 5% for imitation learning.

What they do

Imitation policies trained on a few demos break under clutter and disturbances, and running RL on a real robot is slow and unsafe. RialTo builds a digital twin of the deployment scene, uses sparse-reward RL there to robustify a policy learned from about 15 real demos, and transfers it back to a Franka arm that sees through one depth camera.

How it works

Real-to-sim scenes. The scene is scanned with Polycam, AR Code or NeRFStudio, and users cut out objects and add joints in a GUI that exports a USD for Isaac Sim. All objects get the same default mass and friction.
Inverse distillation. A point-cloud policy trained on the real demos is rolled out in simulation. Its successful runs become 15 sim demos that also record object and joint poses.
RL fine-tuning. PPO with an added BC loss on those demos trains a state-based policy from a sparse success reward, with robot, object and goal poses randomized.
Sim-to-real. The state policy is distilled into a point-cloud policy with DAgger. Training mixes in the 15 real demos and simulated distractor objects.

Results

On book on shelf, RialTo with 15 demos reaches 90/70/60% (pose randomization/distractors/disturbances), against 40/30/20% for BC with 50 demos. The 15 demos take 30 minutes to collect, the 50 take 1 h 45 min.
Co-training on real rather than sim demos gives 3.5x (book on shelf) and 2x (plate on rack) higher success under disturbances.
A policy trained on 4 Objaverse drawers opens the real drawer 10% of the time, against 90% when trained on its scanned twin.
PPO from scratch scores 0% on three of five tasks, against 81-96% when fine-tuned from demos. On drawer opening, RL from point clouds reaches 1% after 35 h, against 96% after 12 h from state.
Six users built an articulated scene in 25 minutes on average, 14 min 40 s of it active work.

Limitations

Depth sensing struggles with thin, transparent and reflective objects. The method covers only articulated rigid objects and quasistatic tasks with slow controllers, and physical parameters are never identified. Training one policy takes about 2 days of wall-clock time.

Videos

Placing a book on a shelf while a person moves the book and the shelf.

Opening a toaster oven in a real kitchen, 3x speed.

Figures

From Figure 1: RialTo overview. 1) Move the real scene into the simulator. 2) Run a policy learned from real demos in... — From Figure 1: RialTo overview. 1) Move the real scene into the simulator. 2) Run a policy learned from real demos in simulation to collect demos with privileged state. 3) Use those demos to guide sparse-reward RL fine-tuning of a state-based policy. 4) Distill it into a point-cloud policy and deploy it in the real world, robust to new distractors, a moved base and perturbations.

From Figure 3: Inverse distillation and RL fine-tuning. 1) Train a point-cloud policy on the real demos with supervis... — From Figure 3: Inverse distillation and RL fine-tuning. 1) Train a point-cloud policy on the real demos with supervised learning. 2) Roll it out on simulated point clouds and keep 15 successful demos that include object poses. 3) Train a state-based policy with sparse-reward RL plus a BC loss on those demos.

From Figure 5: Real-world success of RialTo and imitation learning, both from 15 demos, on six tasks under pose rando... — From Figure 5: Real-world success of RialTo and imitation learning, both from 15 demos, on six tasks under pose randomization, distractors and disturbances. Imitation learning drops sharply once distractors and disturbances are added.

Abstract

Imitation learning methods need significant human supervision to learn policies robust to changes in object poses, physical disturbances, and visual distractors. Reinforcement learning, on the other hand, can explore the environment autonomously to learn robust behaviors but may require impractical amounts of unsafe real-world data collection. To learn performant, robust policies without the burden of unsafe real-world data collection or extensive human supervision, we propose RialTo, a system for robustifying real-world imitation learning policies via reinforcement learning in "digital twin" simulation environments constructed on the fly from small amounts of real-world data. To enable this real-to-sim-to-real pipeline, RialTo proposes an easy-to-use interface for quickly scanning and constructing digital twins of real-world environments. We also introduce a novel "inverse distillation" procedure for bringing real-world demonstrations into simulated environments for efficient fine-tuning, with minimal human intervention and engineering required. We evaluate RialTo across a variety of robotic manipulation problems in the real world, such as robustly stacking dishes on a rack, placing books on a shelf, and six other tasks. RialTo increases (over 67%) in policy robustness without requiring extensive human data collection. Project website and videos at https://real-to-sim-to-real.github.io/RialTo/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong