Accelerated Reinforcement Learning for Soft Robotics: Hours Instead of Years

Roshan S. Kaundinya, Hehui Zheng, Robert K. Katzschmann, George Haller

November 2026

TL;DR A 6,924-parameter reduced-order model of a pneumatic soft arm, fit in 5 minutes from about two hours of hardware data, serves as the RL training environment. The policy trains in 3 hours, deploys without fine-tuning and tracks targets with 40% lower error than policies trained on RNN or transformer models.

What they do

Soft robots lack fast, faithful simulators, and model-free RL on delicate hardware is impractical. This paper trains an RL policy entirely offline on a tiny data-driven model: the arm’s dynamics restricted to a six-dimensional adiabatic spectral submanifold (aSSM), which exists because the arm’s vibrations decay faster than the control actions change. The policy is deployed on a pneumatic soft continuum arm (SoPrA) without fine-tuning.

How it works

Data. A 2.2-hour sweep of 400 random static pressure commands, each held for 20 s, with motion capture recording the tip and mid-section positions.
Model. A cubic polynomial maps static actions to static states. Quadratic polynomials give the manifold parametrization and reduced dynamics, fit by least squares in a delay-embedded observable space.
Policy. Standard policy optimization (Schulman et al., 2017) on parallel model rollouts; the policy sees the current and previous state, the previous action and the next target, and tracks random 2D tip trajectories.
Baselines. The same RL procedure on RNN and transformer world models trained on the same amount of data, plus an MPC controller using the aSSM model.

Results

The aSSM model has 6,924 parameters and takes 5 minutes to learn, against 13 hours for the 4.8M-parameter RNN and 100 hours for the 1.7M-parameter transformer. MLP models diverged on some unseen action sequences.
On hardware lemniscate and square tracks, aSSM-RL has 40% lower tracking error than the best neural-model policy and 20% lower than aSSM-MPC.
The policy runs at 100 Hz, twice the rate of the aSSM-MPC controller, which falls behind on the fastest square.

Limitations

The commanded and measured chamber pressures do not match (about 0.3 s valve lag), which adds wiggles to every closed-loop result and prevented learning the higher-order model correction. The model is only valid when targets move slowly relative to the arm’s decay, and accuracy degrades as targets speed up. Evaluation covers 2D tip tracking with external motion capture and 5 of the 6 chambers active.

Figures

From Figure 1: (a) The SoPrA soft arm. (b) Geometry of the adiabatic spectral submanifold (aSSM) in state and control... — From Figure 1: (a) The SoPrA soft arm. (b) Geometry of the adiabatic spectral submanifold (aSSM) in state and control-action space. (c) Training an RL agent on rollouts of the aSSM-reduced model, then deploying on hardware.

From Figure 5: Pareto plot of RL policy training time against mean closed-loop tip error on hardware, with bubble siz... — From Figure 5: Pareto plot of RL policy training time against mean closed-loop tip error on hardware, with bubble size showing the number of model parameters. The policy trained on the aSSM model is Pareto-dominant over those trained on the RNN and transformer models.

From Figure 6: Mean tip tracking error on the lemniscate and on the square at three speeds, for aSSM-MPC, aSSM-RL, RN... — From Figure 6: Mean tip tracking error on the lemniscate and on the square at three speeds, for aSSM-MPC, aSSM-RL, RNN-RL and Transformer-RL.

From Figure 7: Snapshots of the SoPrA arm during closed-loop tracking of each target, one randomly selected run out o... — From Figure 7: Snapshots of the SoPrA arm during closed-loop tracking of each target, one randomly selected run out of 10 per policy. The path traced by the tip is drawn as a thick line in each policy's color.

Abstract

Reinforcement learning (RL) requires large amounts of interaction data or a reliable simulator, both hard to obtain for nonlinear soft robotic hardware. Recent work shows that a time-scale separation inherent to soft robot dynamics justifies their reduction to low-dimensional invariant manifolds, known as adiabatic spectral submanifolds (aSSMs). We demonstrate that an aSSM-reduced model, a simple polynomial ordinary differential equation learned purely from pre-collected data, can serve as an RL training environment. On a pneumatically actuated soft arm, we train a trajectory-tracking policy in 3 hours on a six-dimensional aSSM-reduced model and deploy it without fine-tuning, whereas reproducing this policy training extrapolates to 3 years of hardware interaction. Our policy tracks dynamic targets with 40% lower error than otherwise identical policies trained on a 4.8-million-parameter recurrent neural network (RNN) or a 1.7-million-parameter transformer, which take 13 and 100 hours to identify, against 5 minutes for our mathematically justified 6,924-parameter model.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong