ManiLadder: Benchmarking Robot Manipulation Through a Categorized and Multi-Difficulty Level Task Ladder
TL;DR A simulation benchmark of 112 manipulation tasks, sorted into four difficulty levels by how well a reference imitation policy learns each from 20 demonstrations. Fine-tuned π0.5 reliably clears only Level 1, and one π0.5 policy trained on all 112 tasks reaches 8.6% success against 33.5% for per-task policies.
What they do
Most simulation benchmarks add breadth, with more scenes and more pick-and-place variants, but do not show how hard a task a method can solve. ManiLadder is a benchmark of 112 tasks in the Genesis simulator, arranged in four difficulty levels. Difficulty is defined empirically: the success rate a standard imitation policy reaches on the task from a fixed amount of data.
How it works
- Tasks. 32 suites of 3-4 tasks cross rigid and deformable objects (rope, cloth, clay, liquid) with four embodiments built from Franka arms and LEAP hands: one or two arms, with grippers or hands. Scripted experts provide 20 multi-view RGB demonstrations per task.
- Calibration. A 50M-parameter flow-matching policy trains on each task’s 20 demonstrations, and its best success rate places the task in a level. Tasks that miss their target range are edited (start-state ranges, object size, tolerances, coordination) and retested, about four rounds per task.
- Checks. Recollecting demonstrations and retraining under three seeds keeps 96.4% of tasks at their level. π0.5 and GPT6-Astra, an agentic policy that writes Python controllers from the demonstrations, put 73.3% and 60.9% of tasks at the same level.
- Metric. Attained Level is the highest level at which at least 80% of tasks, at that level and every lower one, reach 50% (or 80%) success.
Results
- Single-task Attained Level@50 is 1 for the small policy and for π0.5, and 0 for GPT6-Astra. No method wins everywhere: GPT6-Astra solves Stack6Blocks every time, where both imitation policies reach at most 8%.
- One π0.5 trained on all 112 tasks reaches 8.6% overall versus 33.5% for per-task policies, and is worse on all 32 suites.
- Going from 20 to 200 demonstrations raises π0.5 on Level-2 rigid tasks only from 21.4% to 31.8%.
- On 8 real xArm tasks with 50 trials each, success drops level by level as in simulation, from 100% on PickPlacePen to 0% on PingPongTransport and BuildTent.
Limitations
Difficulty is anchored to one reference policy and data budget, and other policies agree on the level of only 61-73% of tasks. Most tasks are tabletop scenes, leaving out scene diversity and navigation by design, and the real-world check covers 8 of 112 tasks.
Figures
Abstract
Recent advances in robotic manipulation have enabled increasingly complex and dexterous behaviors, yet existing benchmarks still focus on simple pick-and-place tasks and lack a systematic framework for measuring progress in robot learning along the axis of task difficulty. We present ManiLadder, a large-scale and organized simulation benchmark for robotic manipulation. ManiLadder contains 112 carefully designed tasks stratified into 4 difficulty levels, with 8 categories in each level, spanning rigid-body and deformable-object manipulation, single-arm and dual-arm settings, and gripper and dexterous-hand end-effectors. To calibrate difficulty, we fix the data amount the same for every task and use a unified flow-matching imitation learning policy under the same training and evaluation protocols, then use its evaluation success rate as an operational empirical difficulty signal. All tasks are iteratively refined until they fall into target difficulty ranges, which are then verified across different random seeds, policies, and agentic learning systems. We further analyze the task properties, compare different policy performances and training settings, analyze task transfer and data scaling behaviors, and conduct real-world experiments showing that the designed tasks preserve their difficulty ordering beyond simulation. ManiLadder provides a calibrated ladder for measuring, comparing, and advancing robotic manipulation capabilities. More results are available at https://maniladder-benchmark.github.io/.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.