Learning Contextual Motion Constraints for Object Rearrangement Exploration

TL;DR From one partial point cloud, MoCo-Net predicts how far an articulated part such as a drawer or door can move before it hits something, and what it would hit. In 1,000 simulated cluttered scenes, a planner using both predictions reaches the target part state in 82.4% of tasks with 0.201 collisions per task, versus 0% and 2.525 when moving the part directly.
What they do
How far a drawer or door can move depends on its joint limits and on whatever stands in its way. MoCo-Net takes a partial point cloud of a cluttered scene and a query point on one movable part, and predicts both the part’s collision-free motion range and a heatmap of where it would collide. A planner uses the two predictions to decide which obstacle or part to move first.
How it works
- Two streams, one encoder. A Point Transformer encodes the scene. One decoder regresses the lower and upper bounds of the feasible range (displacement for prismatic joints, rotation for revolute ones). A second decoder takes the predicted range as an extra input and outputs a collision-likelihood distribution over scene points. Both are trained jointly with an MSE plus KL loss.
- Data. SAPIEN scenes with ShapeNet and PartNet-Mobility articulated objects in random joint states and 0-15 random obstacles. Ground truth comes from moving each part in simulation until it collides.
- Planning. If the target amount lies inside the predicted range, the part is moved. Otherwise the planner moves the object at the heatmap’s most likely collision point, recursing when that object is itself blocked (a trash can blocking a dishwasher door that blocks a drawer).
Results
- Trained on 16,000 scenes, tested on 5,000 new layouts and 5,000 scenes with unseen objects. On unseen objects, 60.0% of range predictions fall within 10% of ground truth, versus 44.5% for simulating motion on the partial point cloud with ground-truth joints, and 75.7% of collision points are detected.
- Joint training beats single-task models, and feeding the predicted range into the heatmap branch works better than the reverse.
- Planning in 1,000 scenes: 82.4% success with 5.47 rearrangement actions and 0.201 collisions on average, versus 80.0%, 6.19 and 0.235 for a greedy nearest-obstacle baseline and 0%, 6.06 and 2.53 for direct manipulation.
Limitations
All experiments are in simulation, with no real sensor data or robot. The planning results count rearrangement actions; how a robot would grasp and execute each one is not described. The paper shows failure cases where limits are misjudged or obstacles missed, and 40% of range predictions on unseen objects fall outside the 10% tolerance.
Figures




Abstract
The motion of an articulated part in cluttered surroundings is constrained not only by an object's intrinsic joint limits but also by extrinsic obstacles. However, existing approaches primarily consider object articulation properties alone. Here we present Motion Constraint Prediction Network (MoCo-Net), a two-stream neural network that jointly predicts the continuous motion ranges of articulated parts and obstacle heatmaps highlighting potential collision regions from partial-view point cloud observation. By capturing both internal and external motion constraints, MoCo-Net produces more accurate feasible motion range prediction and obstacles recognition, improves generalization for novel objects in new scenes, and enables robots to construct a scene configuration space for efficient exploration of rearrangement actions. Our experiments show that MoCo-Net's two-stream design enhances each other to deliver more accurate predictions, which subsequently form a compact configuration space for a customized planner to efficiently rearrange the scene.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
