Augmentron: Scalable Multi-View Visual Augmentation for Robot Learning
TL;DR Augmentron restyles the background and table of real multi-camera robot demonstrations by rendering generated scene images through the recorded camera poses, so the edits stay consistent across views and time. Replacing half of a pi0.5 fine-tuning set with its data raises real-robot success with unseen distractors from 18.0% to 45.3%.
What they do
Robot demonstrations tend to repeat the same scenes, so policies break when task-irrelevant visuals change. A pi0.5 policy fine-tuned on their real data drops from 66.7% to 18.0% success once distractor objects are added. Augmentron is an augmentation pipeline that generates one reusable scene representation per demonstration and renders it into every camera and timestep, instead of running a generative model on each frame.
How it works
- Masks. The robot mesh, placed at each recorded joint configuration, is projected into each calibrated camera. A language model turns the task annotations into prompts for SAM 3, which segments the task objects and the table surface.
- Scene representation. A per-pixel temporal median over frames, ignoring masked pixels, gives a nearly foreground-free reference image. FLUX edits it into a top-down scene image and a six-face cubemap.
- Rendering. Fixed cameras reuse one rendered image. Wrist cameras render the cubemap (environment) or a texture on the calibrated table plane (workspace) from each recorded pose. The original robot and object pixels go back on top, and actions and labels are unchanged.
Results
- Three real tasks (cup on saucer, towel folding, flower into vase) with 1,793 demonstrations (9.16 h). Each task and condition gets 50 physical rollouts.
- At equal data volume, replacing half of the real data with Augmentron data raises pooled success from 66.7% to 76.7% in distribution, from 18.0% to 45.3% with distractors, and from 50.7% to 67.3% on a steel instead of wooden table.
- At 45k steps, Augmentron reaches 71.3/66.0/60.7% against 57.3/53.3/41.3% for RoboEngine, though it uses 3x the data volume against 2x.
- Four-camera demonstrations take 0.93x their video duration to process, against 16.28x for RoboEngine.
Limitations
There is one checkpoint and seed per task and recipe, so training variance is unmeasured. The baseline comparison does not match data volume or isolate components. A larger augmented fraction did not help uniformly: vase insertion with distractors fell from 32% to 12%.
Figures
Abstract
Robot policies trained on visually narrow demonstrations can become sensitive to environmental features that are unrelated to the task. Existing generative augmentation methods can introduce visual diversity, but processing every frame and camera view independently is computationally expensive and may produce inconsistent observations across cameras and timestamps. We introduce Augmentron, a geometry-aware pipeline that generates reusable scene representations and renders them through recorded camera poses. This produces temporally and multi-view consistent environment and workspace variations while preserving the robot, task objects, actions, and annotations. Across three physical manipulation tasks, replacing half of the real world training dataset with Augmentron data improves model success across three tasks for both in out-of-distribution distractor objects, and out-of-distribution workspace variation evaluations. In a separate comparison against alternate augmentation techniques, Augmentron achieves the highest pooled success in all three conditions. The results show that introducing augmented data to a data training recipe can help policies generalize to dynamic real-world settings.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.