Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning
TL;DR Isaac Gym keeps the whole RL loop on one GPU, with physics, observations, rewards and PPO training all on the device, and simulator state exposed as PyTorch tensors that never pass through the CPU. On a single A100 it trains Shadow Hand cube reorientation to 20 consecutive successes in under 1 hour; OpenAI needed 30 hours on 6144 CPU cores plus 8 V100 GPUs.
What they do
Robot RL usually simulates physics on CPUs and runs the networks on GPUs, so state has to be copied back and forth at every step. Isaac Gym keeps the whole loop on one GPU. PhysX steps tens of thousands of parallel environments, observations and rewards are computed on PyTorch tensors, and the policy’s actions are written straight back into the simulator’s buffers.
How it works
- Tensor API. Simulator state (root, DOF and rigid-body states, contact forces, Jacobians, mass matrices) is held in flat GPU buffers that are wrapped as PyTorch tensors without copying. Forces, torques and PD targets go in the same way. An index buffer resets only a subset of environments.
- Physics. PhysX reduced-coordinate articulations solved with Temporal Gauss-Seidel, plus tendons for the Shadow Hand. Assets load from URDF or MJCF, and most properties can be changed mid-run for domain randomization.
- Training. Every task trains with GPU-vectorized PPO (rl_games). Shadow Hand and TriFinger use an asymmetric actor-critic whose value function gets privileged state.
Results
- On a single A100, Ant reaches 3000 reward in 20 s, Humanoid runs in under 4 min and ANYmal tracks velocity commands in under 2 min. AMP character animation takes about 6 min, against about 30 h on 16 CPU cores in PyBullet.
- Shadow Hand with OpenAI observations and domain randomization: 20 consecutive successes in under 1 h (feed-forward) and 37 in under 6 h (LSTM). OpenAI needed 30 h and 17 h.
- Throughput reaches 700K environment steps/s for Ant, 200K for Humanoid and 150K for Shadow Hand.
- Sim-to-real: an ANYmal rough-terrain policy is trained on an RTX A6000 and transferred to the real robot in under 20 min. TriFinger cube reposing succeeds 55% of the time on the real robot.
Limitations
The Shadow Hand is shown in simulation only, and its randomization leaves out action delay and motor backlash. The comparison with OpenAI also differs in other ways: continuous actions, a different reward, and 5 seeds instead of 1. With a fixed experience budget, more environments means a shorter PPO horizon. Past 4096 environments, Humanoid trained more slowly and learned worse gaits.
Videos
Figures
Abstract
Isaac Gym offers a high performance learning platform to train policies for wide variety of robotics tasks directly on GPU. Both physics simulation and the neural network policy training reside on GPU and communicate by directly passing data from physics buffers to PyTorch tensors without ever going through any CPU bottlenecks. This leads to blazing fast training times for complex robotics tasks on a single GPU with 2-3 orders of magnitude improvements compared to conventional RL training that uses a CPU based simulator and GPU for neural networks. We host the results and videos at \url{https://sites.google.com/view/isaacgym-nvidia} and isaac gym can be downloaded at \url{https://developer.nvidia.com/isaac-gym}.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.