Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Benchmarking Real-World Robotic Manipulation through Stochastic Challenges

Elias Gläßer, Gülhan Şikaroğlu, Pavlo Kupyn, Marc Pollefeys, Zuria Bauer, Hermann Blum

November 2026

TL;DR Instead of a fixed test set, a real-world benchmark standardizes a seeded distribution of scenes, so labs with different robots can be compared statistically. On randomized IKEA KALLAX shelves, 568 trials in two labs separate a Spot from a Stretch 3 on door opening (71% against 18%) and drawer opening (88% against 64%).

What they do

Real-world manipulation results are hard to compare across labs: fixed object sets invite overfitting, and existing distributed benchmarks fix the robot or the task and stay on the tabletop. This paper standardizes a distribution of scenes instead. A seed defines a full physical scene that any lab can rebuild, and robots are compared by success rate over sampled seeds, with a statistical framework that says how many seeds a comparison needs.

How it works

  • KALLAX benchmark. A seeded randomizer fills a 3x4 IKEA KALLAX shelf: half the positions are left empty and the rest get inserts such as doors and drawers from a pool of up to 82 options, with 79 YCB objects for search tasks. A website generates configurations, assembly instructions and task lists, and records the reported outcomes.
  • Variance decomposition. The variance of the benchmark score splits into trial noise, which shrinks with repeated attempts, and scene difficulty, which shrinks only with more seeds. This gives the number of seeds needed to detect a given performance gap.
  • Shared seeds. Testing two robots on the same seeds cancels scene difficulty; mixed designs share only a fraction of the seeds.

Results

  • 568 trials on door and drawer opening (success means opened by at least 10 cm), with a Boston Dynamics Spot and a Hello Robot Stretch 3 in two labs in different countries.
  • Spot beats Stretch by 0.531 on doors (0.711 against 0.180; detectable margin 0.333) and by 0.244 on drawers (0.879 against 0.635; margin 0.232), both distinguishable at alpha = 0.10 and 80% power.
  • Sharing half the seeds lowers the detectable margins to 0.256 and 0.183. With small random subsets of seeds, the apparent ranking on drawers can reverse.

Limitations

Many trials are needed for a detectable margin: the drawer comparison sits near the decision boundary, and halving the margin takes about four times as many seeds. One researcher rebuilt the shared scenes in both labs, so variability between reconstruction teams is not measured. Success is self-reported, and only door and drawer opening were evaluated.

Figures

From Figure 1: Stochastic Challenges target several desired aspects of robotic benchmarking: system-level innovation,...
From Figure 1: Stochastic Challenges target several desired aspects of robotic benchmarking: system-level innovation, i.e. not restricting evaluation to specific hardware; local evaluation, avoiding the effort to transport systems; and reduced overfitting, testing true generalization instead of pre-known tasks.
From Figure 2: Concept overview. Seeded randomization generates reproducible KALLAX manipulation scenes; robots acros...
From Figure 2: Concept overview. Seeded randomization generates reproducible KALLAX manipulation scenes; robots across labs (Stretch 3 and Spot) execute sampled configurations, with outcomes statistically aggregated for robust comparison.
From Figure 3: Experimental setup. The KALLAX benchmark draws sample configs from the same distribution, which are th...
From Figure 3: Experimental setup. The KALLAX benchmark draws sample configs from the same distribution, which are then recreated in two different labs to evaluate the two different robotic systems.
Abstract

A challenging bottleneck in manipulation research is reproducibility and comparability of methods in real-world environments. Since manipulation is an interactive task, it cannot be evaluated on a static dataset of observations. Benchmarking in simulation is a common remedy, but cannot cover the whole complexity of real-world tests. This paper proposes a new paradigm of stochastic challenges. Like the well-known YCB objects, it is based on reproducible real-world scenes and therefore allows system-level innovation and comparability between different platforms. Due to their stochastic nature, our challenges however, mimic the rigor and surprise of hidden test sets. To this end, we show how different robots in different labs can be compared based on their success rate over different random task samples, if these samples come from the same distribution. We propose stochastic challenges as a general paradigm for robot benchmarking and implement this paradigm in a novel benchmark for mobile manipulation, on which we compare two robots tested in two different labs.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.