Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting

Mohammad Nomaan Qureshi, Sparsh Garg, Francisco Yandun, David Held, George Kantor, Abhisesh Silwal

Seminal September 2024

Real UR5 pushing the T-block on the real table.

TL;DR SplatSim swaps a simulator's mesh renderer for a Gaussian Splat of the real scene, moving each robot link and object's Gaussians to follow the PyBullet state. RGB diffusion policies trained only on these renderings transfer zero-shot to a real UR5 with 86.25% average success over four tasks, against 97.5% for policies trained on real data.

What they do

RGB policies trained in simulation rarely transfer because simulated images look nothing like real ones. SplatSim keeps the simulator (PyBullet) for physics but replaces its mesh rendering with a Gaussian Splat of the real deployment scene. Demonstrations generated in simulation are rendered photorealistically and used to train RGB policies that run on the real robot without finetuning. The only real-world input is a capture of the static scene with the robot.

How it works

  • Robot splat. The robot’s Gaussians are segmented by hand and aligned to the simulator robot with ICP. CAD bounding boxes split them into links, and PyBullet forward kinematics moves each link’s Gaussians to any joint configuration.
  • Gripper and objects. Gripper links don’t fit axis-aligned boxes, so a KNN classifier trained on labeled simulator point clouds assigns each Gaussian a link. Separately captured object splats are aligned to their simulated point clouds with ICP and moved with the simulated object pose.
  • Policy. Demonstrations come from Gello teleoperation (T-Push) or a privileged-state motion planner (the other tasks). A Diffusion Policy takes the rendered images plus end-effector pose, trained with noise, random erasing and brightness/contrast augmentations.

Results

  • On a UR5 with a Robotiq 2F-85 gripper and two RealSense D455 cameras, zero-shot Sim2Real success is 90% T-Push, 95% Pick-Up-Apple, 90% Orange-on-Plate and 70% Assembly (40 trials each), averaging 86.25% versus 97.5% Real2Real and 95.62% Sim2Sim.
  • Demonstration collection took 3 h of human effort in simulation versus 20.5 h in the real world.
  • Without augmentations, average Sim2Real success falls to 21%.
  • Rendered robot images over 300 joint configurations reach 22.62 PSNR and 0.7845 SSIM against real photos.

Limitations

Rigid bodies only: cloth, liquids and plants are out of scope. The renderings lack shadows and changing reflections, and flexible parts such as robot cables render incorrectly, which is why augmentation is needed. The robot’s Gaussians are segmented from the scene by hand.

Videos

Teleoperating the simulated robot with a Gello arm while SplatSim renders the scene.
SplatSim renderings of simulated robot-object trajectories in the scanned scene.

Figures

From Figure 1: Gaussian Splatting used as the rendering primitive inside an existing simulator. Splats of the robot a...
From Figure 1: Gaussian Splatting used as the rendering primitive inside an existing simulator. Splats of the robot and object are rendered at novel joint angles and object positions to produce photorealistic training data.
From Figure 2: Top, expert demonstrations collected in PyBullet (Gello teleoperation or a privileged motion planner) ...
From Figure 2: Top, expert demonstrations collected in PyBullet (Gello teleoperation or a privileged motion planner) are replayed through simulator-aligned splat models to render RGB observations, which train a diffusion policy together with the end-effector pose. Bottom, the frozen policy is deployed directly on the real robot.
From Figure 3: The scene is splatted with the robot at its home position. The robot Gaussians are manually segmented ...
From Figure 3: The scene is splatted with the robot at its home position. The robot Gaussians are manually segmented and aligned to the simulator robot with ICP, then split into links so forward kinematics can render the robot at any joint configuration.
From Figure 5: SplatSim renderings over time for the four tasks: T-Push, Pick-Up-Apple, Orange-on-Plate and Assembly.
From Figure 5: SplatSim renderings over time for the four tasks: T-Push, Pick-Up-Apple, Orange-on-Plate and Assembly.
Abstract

Sim2Real transfer, particularly for manipulation policies relying on RGB images, remains a critical challenge in robotics due to the significant domain shift between synthetic and real-world visual data. In this paper, we propose SplatSim, a novel framework that leverages Gaussian Splatting as the primary rendering primitive to reduce the Sim2Real gap for RGB-based manipulation policies. By replacing traditional mesh representations with Gaussian Splats in simulators, SplatSim produces highly photorealistic synthetic data while maintaining the scalability and cost-efficiency of simulation. We demonstrate the effectiveness of our framework by training manipulation policies within SplatSim and deploying them in the real world in a zero-shot manner, achieving an average success rate of 86.25%, compared to 97.5% for policies trained on real-world data. Videos can be found on our project page: https://splatsim.github.io

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.