An Empirical Study of Reinforcement Learning for Keypoint Positioning of Deformable Linear Objects

Vinay Rodrigues, Sara Zeynalpour, Gokhan Solak, Sebastian Hjorth, Arash Ajoudani

November 2026

TL;DR PPO policies trained in GPU-parallel MuJoCo Playground move a rope held by a Franka arm so that its tip, or its tip and then its grasped end, reaches targets on a table. Trained with noise and domain randomization and fed markerless RGB-D keypoints, they transfer zero-shot and succeed in 9/10 single-point, 10/10 disturbance-recovery and 8/10 two-point real trials.

What they do

A study of model-free RL (PPO) for planar manipulation of a deformable linear object (DLO) whose one end is grasped by a Franka Panda. The policy moves the end effector so that keypoints on the rope reach table targets: the free tip alone (single-point), or the tip and then the grasped end (two-point).

How it works

Training stacks. Stable-Baselines3 and its JAX port SBX, both with CPU MuJoCo and 4 environments, against MuJoCo Playground, which runs JAX PPO and MJX physics on the GPU across 1024 environments, each with its own hyperparameter sweep.
Observations. The actor sees 5 rope keypoints relative to the end effector, robot proprioception and target errors, and outputs planar translation and yaw increments at 10 Hz. An asymmetric critic also gets 12 keypoints and other privileged simulator state.
Robustness training. Observation noise, up to one step of delay, randomized friction and robot dynamics, and a random external force on the rope for disturbance recovery.
Perception. A fine-tuned YOLO26-seg model segments the rope in overhead RealSense images; the mask is skeletonized into 5 keypoints and projected onto the table plane.

Results

In about 240 min of wall-clock training, MuJoCo Playground reaches 0.95 final success against 0.86 (SB3) and 0.88 (SBX), collecting about 13x more interactions.
End-effector-relative keypoints and the asymmetric critic each learn faster than robot-frame keypoints or a symmetric critic.
Simulation with noise and randomization (250 episodes each): 96.4% single-point, 98.0% with disturbances, 88.8% two-point. Two-point failures cluster near the workspace boundary (19.8% failure rate vs 4.3% in the interior).
Real robot, silicone catheter, zero-shot: 9/10, 10/10 and 8/10, with median completion times of 6.55-9.30 s.

Limitations

Slow, quasi-static planar motion with one end always grasped, and a separate policy for each number of tracked points. Bending and torsional stiffness were not randomized, and the multi-length policy was tested only on its three training lengths. Real tests used one catheter and 10 trials per task. Failures came from the robot occluding the rope, the catheter snapping back after sharp bends, and perception delay causing overshoot.

Figures

From Figure 2: Closed-loop deployment and asymmetric actor-critic training. Rope keypoints from the vision system, ro... — From Figure 2: Closed-loop deployment and asymmetric actor-critic training. Rope keypoints from the vision system, robot state and task targets feed the PPO policy, which drives a Cartesian pose controller.

From Figure 6: Simulation success rates for single- and two-point tracking. Open circles are the five evaluation seed... — From Figure 6: Simulation success rates for single- and two-point tracking. Open circles are the five evaluation seeds; diamonds and error bars are pooled estimates with 95% Wilson intervals. The disturbance condition was evaluated only for single-point tracking.

From Figure 9: Snapshots of a successful real-world two-point trial. The green and red points are the targets for the... — From Figure 9: Snapshots of a successful real-world two-point trial. The green and red points are the targets for the grasped-end and tip keypoints.

Abstract

Deformable linear object (DLO) manipulation is challenging because high-dimensional configurations, nonlinear contact dynamics, and diverse material and geometric properties produce a broad range of object responses that are both difficult to model and data-intensive to learn. This work studies reinforcement learning for planar manipulation of a one-end-grasped DLO, focusing on training efficiency, observation design, robustness, and zero-shot sim-to-real transfer. We compare three Proximal Policy Optimization training stacks and find that the GPU-parallel MuJoCo Playground stack enables faster convergence and more consistent performance than CPU-based alternatives under the same wall-clock budget. Relative keypoint observations and an asymmetric actor-critic formulation further improve learning efficiency. Policies trained with observation noise, delay, domain randomization, and disturbances achieve simulation success rates of 96.4% for single-point tracking, 98.0% for disturbance recovery, and 88.8% for two-point tracking. Using markerless RGB-D keypoint estimation, the policies transfer without real-world fine-tuning, succeeding in 9/10, 10/10, and 8/10 physical trials, respectively.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong