Lazy-MOSAIC: Accelerating Skill-Centric Manipulation Planning with Learned Skill-Effect World Models

TL;DR Lazy-MOSAIC speeds up a skill-centric manipulation planner by searching over skill outcomes predicted by a cheap learned model and simulating only the most promising skills, while every skill in a returned plan is still validated in physics simulation. On simulated cluttered-shelf retrieval it solves 104 of 160 trials versus 81 for MOSAIC, with 64-69% fewer simulator queries and 2.1-6.6x faster planning.
What they do
MOSAIC plans long-horizon manipulation by letting skills drive a non-sequential search: generator skills propose behaviors where they are likely to succeed, and connector skills link them. It simulates every candidate skill before adding it to the graph, which takes over 80% of its planning time. Lazy-MOSAIC searches over outcomes predicted by a cheap learned model instead and simulates only skills on promising candidate plans.
How it works
- Two graphs. An optimistic graph holds proposals scored by the learned model (predicted success, end state and uncertainty); a certified graph holds skills already rolled out in simulation. A shortest-path search runs over both, with costs that penalize low predicted success and high uncertainty.
- Certification and repair. The first uncertified skill on the best path is simulated from its realized start state. If the outcome differs from the prediction, only the dependent proposals that no longer fit are removed, and the realized state stays in the graph to build on. A plan is returned only when every skill in it is certified.
- Skill-effect model. One ensemble of three small Deep Sets networks covers all five skills (pick, place, move, push, pull) and any number of objects. It is trained on about 35,000 logged proposals and scores 128 of them in about 0.7 ms.
Results
- Three shelf-retrieval tasks with a simulated UR10e, eight difficulty settings, 20 trials each: Lazy-MOSAIC solves 104/160 vs 81/160 for MOSAIC. On successful runs it averages 43, 83 and 216 s on easy, medium and hard settings vs 124, 331 and 479 s.
- Simulator queries drop from 46.4/67.9/134.5 to 16.9/21.7/41.3. Ranking proposals with the model but simulating right away gives only 44.3/62.9/107.9, so most of the saving comes from deferral.
- Executing the predicted plan without certification reaches the goal in only 33/160 trials; dropping the uncertainty term lowers success to 90/160. On the hardest wrench task Lazy-MOSAIC solves 4/20, and MOSAIC and the other baselines solve none.
Limitations
The learned model is trained offline for one known workspace and set of object types; generalizing across workspaces is left open. Its accuracy limits the gain from laziness, and maintaining and repairing the optimistic graph costs more as it grows, which partly explains the smaller speedup on the hardest settings. All quantitative results are in simulation; the physical robot appears only in a demonstration video.
Figures



Abstract
Long-horizon manipulation tasks require a robot to perform a sequence of physical interactions with its environment, typically combining several different skills. Planning such behaviors efficiently is challenging: given a library of parameterized skills, solutions lie in a vast hybrid space of skill sequences and their parameters, and evaluating each candidate skill's feasibility and effect with a high-fidelity world model such as a physics simulator is computationally expensive. MOSAIC, a recent skill-centric approach, lets the skills guide a non-sequential search toward the regions of this space where they are likely to succeed; however, it pays the full cost of simulation for every parameterized skill it considers. In this work, we introduce Lazy-MOSAIC, a lazy planning framework that instead searches over hypothesized skill outcomes. A cheap learned skill-effect world model predicts, in a fraction of a millisecond, a skill's feasibility and the world state it produces. Lazy-MOSAIC then uses these predictions to focus expensive simulation on promising candidate skills and, when a hypothesis is invalidated, repairs only the affected part of the search, while ensuring that every skill in a returned plan is validated in simulation. We evaluate Lazy-MOSAIC on cluttered-shelf retrieval tasks of increasing difficulty and demonstrate it on a physical robot, showing that it plans substantially faster than baselines with fewer simulator queries, while matching or exceeding their success rate in every setting. Visit skill-mosaic.github.io.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
