CrowdMAPS: Crowdsourcing Multiple Actions per State for Real-World Imitation Learning

Yilong Song, Harshvardhan Chandirasekar, Karthik Desingh, Chen Zhu-Tian

November 2026

Physical data collection by teleoperation, which records one action per visited state.

TL;DR Remote non-experts use a web 3D interface to add alternative valid actions to states a real robot already visited, giving up to ten action labels per state. Going from one to ten labels per state raises keypose-policy success from 38.7% to 74.2% across five real-robot tasks.

What they do

A teleoperated demonstration records only the one action that was executed at each state, although many states admit several valid next actions. CrowdMAPS keeps the real robot for collecting states and crowdsources extra action labels for those states in software. Remote annotators, most with no robot-arm experience, add alternative actions through a browser, with no access to the hardware.

How it works

Keyframe goal poses. Actions are end-effector goal poses plus a gripper command, executed by a motion planner. Keyframes, at the start and at each goal completion, carry the executed action and the crowd alternatives.
Annotation interface. For each recorded keyframe, LangSAM and Any6D place CAD meshes of the task objects in Isaac Lab. Annotators drag a rendered robot to a goal pose and can simulate it before submitting. A researcher accepts or rejects each submission.
Training. Keypose diffusion policies are trained on all labels per state, with executed actions weighted above alternatives. For trajectory policies, goal poses are interpolated into action chunks and co-trained with teleoperated demonstrations.

Results

47 annotators; 78.7% had never used a robot arm. Median submission time was 7-12 s, and 51-98% of submissions were accepted, depending on the task.
Real WidowX arm, five tasks, 25 rollouts per model. Going from 1 to 10 labels per state lifts average keypose success from 38.7% to 74.2%. A Gaussian-noise baseline with the same density reaches 43.3%.
In trajectory policies, 10-label MAPS data beats adding more expert teleoperation on all five tasks (Sorting 88% vs. 52%) for similar robot time (2.2 vs. 2.3 h). At matched performance it costs about 1.5x more total human time.

Limitations

Goal-pose actions are less expressive than dense teleoperation, and only quasi-static, single-arm tabletop tasks were tested. Each new task needs CAD models and object descriptions, and a researcher validates every submission. Alternative actions are never executed, so the states they lead to are missing from training (MAPS shift).

Videos

Remote users specify alternative valid actions in the web interface.

Figures

From Figure 1: After standard real-robot data collection (a), CrowdMAPS (b) sends the visited states to remote annota... — From Figure 1: After standard real-robot data collection (a), CrowdMAPS (b) sends the visited states to remote annotators, who provide unexecuted alternative actions through a web interface. The result is real-robot Multiple Actions per State (MAPS) data (c).

From Figure 3: Recorded real-robot keyframes (a) contain RGB-D and proprioceptive data. LangSAM segments the task obj... — From Figure 3: Recorded real-robot keyframes (a) contain RGB-D and proprioceptive data. LangSAM segments the task objects, Any6D estimates their 6-DoF poses, and their CAD meshes are placed in Isaac Lab to reconstruct the scene. In the annotation interface (b), annotators move the rendered robot to specify an alternative goal-pose action.

From Figure 4: Keypose-policy success rate against the number of training episodes on Insertion, Pouring, Sorting and... — From Figure 4: Keypose-policy success rate against the number of training episodes on Insertion, Pouring, Sorting and Switches-, for MAPS widths K = 1, 2, 5 and 10 actions per state. Each point is 25 real-world rollouts with randomized initializations.

From Figure 7: Executed and crowd-submitted end-effector goal poses, projected onto one keyframe from each task. Reje... — From Figure 7: Executed and crowd-submitted end-effector goal poses, projected onto one keyframe from each task. Rejected actions are semantically wrong or physically incoherent; accepted ones cover variation both within one mode and across modes.

Abstract

Real-world robot imitation learning is often limited by the cost of collecting physical demonstrations, which typically yield only one action label per visited state. We present CrowdMAPS, a web-based crowdsourcing paradigm for augmenting real-robot datasets with Multiple Actions per State (MAPS). After or during standard robot data collection, CrowdMAPS distributes visited robot states to remote annotators through a web-based 3D interface, where they specify alternative valid actions without requiring access to physical robot hardware. This expands robot datasets along a complementary "width" dimension--by increasing the number of actions per visited state--rather than only by collecting more trajectories. We evaluate CrowdMAPS using keyframe-centric goal-pose annotations supplied by non-expert annotators and train imitation policies on the resulting MAPS data. Across five real-world manipulation tasks, increasing MAPS width from one to ten actions per state improves sample efficiency and increases average success rates by 35.6 percentage points for keypose policies and 33.0 percentage points for trajectory policies. These results show that software-mediated human annotations can augment physical robot data to afford denser supervision, establishing CrowdMAPS as a scalable digital layer for real-robot data collection.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong