Habitat 2.0: Training Home Assistants to Rearrange their Habitat
TL;DR Habitat 2.0 is a physics-enabled home simulator with an interactive apartment dataset (ReplicaCAD) and a rearrangement benchmark for a Fetch robot, running at 25,734 steps per second on one 8-GPU node (850x real-time). On its tasks, flat RL fails at long-horizon rearrangement, a hierarchy of RL skills does better, and sense-plan-act is more brittle.
What they do
Habitat 2.0 is a simulation stack for training mobile manipulators on household rearrangement. It has three parts: ReplicaCAD, an artist-built interactive recreation of the Replica ‘FRL-apartment’ (92 objects, 105 furniture layouts, an articulated fridge and drawers); the H2.0 simulator, which gives up breadth (no deformables, fluids, audio or touch) for speed; and the Home Assistant Benchmark (Tidy House, Prepare Groceries, Set Table), where a Fetch robot gets only the start and goal 3D position of each object.
How it works
- Lazy physics. Distant objects are not updated, the base moves kinematically on a navmesh, and Bullet physics and Magnum rendering are integrated in C++.
- Interleaved physics and rendering. The policy acts on the observation from one step earlier, so the CPU computes the next state while the GPU renders the current view.
- Simplify and reuse. Compressed textures (4-6x less GPU memory), cached assets for fast resets, and convex collision meshes.
- Baselines. Grasping is abstracted: an object within 15 cm snaps into the gripper. One flat RL policy (MonolithicRL) is compared with a STRIPS task planner chaining RL skills (TP+SRL) or RRTConnect motion planning (TP+SPA).
Results
- Speed: 1,191 steps/s in one process (iGibson reports 100), 8,186 on one GPU and 25,734 on 8 GPUs, each step rendering RGB-D and simulating 1/30 s.
- Pick: MonolithicRL reaches 86.3% on unseen layouts versus 72.7% for SPA and 80.0% for SPA with ground-truth geometry, but drops to 52.7% on unseen receptacles (SPA 60.3%).
- A blind policy using only proprioception still picks 78% of the time.
- On the benchmark tasks MonolithicRL “performs abysmally”; the hierarchical methods lose success as the number of interactions grows, and TP+SRL beats TP+SPA on Prepare Groceries, where the cluttered fridge is hard to perceive.
Limitations
Simulation only, with abstracted grasping. Chained skills suffer hand-off problems, such as navigating to a spot where the next manipulation fails. Opening unknown containers with SPA is left out of scope, so Set Table has no SPA baseline. ReplicaCAD models apartments from one country (USA), and full training is bottlenecked by environment synchronization and asset reloading on reset.
Videos
Figures
Abstract
We introduce Habitat 2.0 (H2.0), a simulation platform for training virtual robots in interactive 3D environments and complex physics-enabled scenarios. We make comprehensive contributions to all levels of the embodied AI stack - data, simulation, and benchmark tasks. Specifically, we present: (i) ReplicaCAD: an artist-authored, annotated, reconfigurable 3D dataset of apartments (matching real spaces) with articulated objects (e.g. cabinets and drawers that can open/close); (ii) H2.0: a high-performance physics-enabled 3D simulator with speeds exceeding 25,000 simulation steps per second (850x real-time) on an 8-GPU node, representing 100x speed-ups over prior work; and, (iii) Home Assistant Benchmark (HAB): a suite of common tasks for assistive robots (tidy the house, prepare groceries, set the table) that test a range of mobile manipulation capabilities. These large-scale engineering contributions allow us to systematically compare deep reinforcement learning (RL) at scale and classical sense-plan-act (SPA) pipelines in long-horizon structured tasks, with an emphasis on generalization to new objects, receptacles, and layouts. We find that (1) flat RL policies struggle on HAB compared to hierarchical ones; (2) a hierarchy with independent skills suffers from 'hand-off problems', and (3) SPA pipelines are more brittle than RL policies.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.