Towards Expedited Impedance Tuning of a Robotic Prosthesis for Personalized Gait Assistance by Reinforcement Learning Control

Minhan Li, Yue Wen, Xiang Gao, Jennie Si, He Helen Huang

Seminal June 2020

TL;DR PICE tunes the 12 impedance parameters of a powered knee prosthesis with policy iteration over a positive-semidefinite quadratic value function, trained offline on logged data and refined online with the user walking. Starting from an offline pre-trained policy cut tuning by 58 impedance updates (about 7 minutes of walking) on average compared with a random start.

What they do

Powered knee prostheses usually run a finite-state impedance controller: four gait phases, each with a stiffness, equilibrium angle and damping, so 12 parameters that prosthetists hand-tune for hours per user. PICE (policy iteration with constraint embedded) learns tuning policies that adjust these parameters until the prosthetic knee’s motion matches able-bodied targets, and is built to need little human walking time.

How it works

Problem. Each phase has its own policy. The state is the peak-angle error and duration error of that phase’s knee feature against the target; the action is the change in the three impedance parameters; the stage cost is quadratic in both.
Value function. A quadratic Q-function constrained to be positive semidefinite, so approximation error cannot produce negative costs. Policy evaluation solves a projected Bellman equation iteratively; policy improvement is a small QP.
Offline and online. Offline training reuses logged samples off-policy with importance sampling; online training uses on-policy batches of 15 samples and stops early once the cost is falling or small.
Safety. Impedance is updated every 4 gait cycles; if a peak error leaves ±12 deg, parameters reset to a known safe set.

Results

Subjects: two able-bodied, two transfemoral amputees. Offline training on 105 samples converged in 6 policy updates (10 s of computation). With 135 logged samples, all four phases met the success criterion with no online updates.
Pre-trained initial policies needed online updates in 1 phase versus all 4 for random ones, and 28-43 impedance updates versus 53-126; on average 58 fewer, about 7 minutes of walking.
Policies transferred: a treadmill policy tuned level-ground walking in 46 updates and a 4-deg ramp in 30 with no policy updates; an able-bodied subject’s policy worked for an amputee after 3 policy updates in one phase.

Limitations

Only four subjects and treadmill, level-ground and ramp walking. The four feature points leave the rest of the knee profile uncontrolled, and the able-bodied target profile may not match user preference or other gait measures such as symmetry or energy cost.

Figures

From Figure 1: The two loops. In the impedance update loop (red), a tuning policy adjusts the impedance parameters of... — From Figure 1: The two loops. In the impedance update loop (red), a tuning policy adjusts the impedance parameters of a four-phase finite-state impedance controller (stance flexion, stance extension, swing flexion, swing extension) from the measured knee kinematics; in the policy update loop (blue), PICE updates that policy.

From Figure 2: Knee kinematic features in one gait cycle: a peak angle P and a duration D for each of the four phases.

From Figure 3: The robotic knee prosthesis prototype: a motor and ball screw drive a slider-crank, with a potentiomet... — From Figure 3: The robotic knee prosthesis prototype: a motor and ball screw drive a slider-crank, with a potentiometer at the knee and a load cell on the pylon.

From Figure 5: Prosthetic knee angle over the gait cycle with the initial and the final tuned impedance parameters, i... — From Figure 5: Prosthetic knee angle over the gait cycle with the initial and the final tuned impedance parameters, in four online trials started from pre-trained policies, against the target points.

Abstract

Personalizing medical devices such as lower limb wearable robots is challenging. While the initial feasibility of automating the process of knee prosthesis control parameter tuning has been demonstrated in a principled way, the next critical issue is to improve tuning efficiency and speed it up for the human user, in clinic settings, while maintaining human safety. We, therefore, propose a policy iteration with constraint embedded (PICE) method as an innovative solution to the problem under the framework of reinforcement learning. Central to PICE is the use of a projected Bellman equation with a constraint of assuring positive semidefiniteness of performance values during policy evaluation. Additionally, we developed both online and offline PICE implementations that provide additional flexibility for the designer to fully utilize measurement data, either from on-policy or off-policy, to further improve PICE tuning efficiency. Our human subject testing showed that the PICE provided effective policies with significantly reduced tuning time. For the first time, we also experimentally evaluated and demonstrated the robustness of the deployed policies by applying them to different tasks and users. Putting it together, our new way of problem solving has been effective as PICE has demonstrated its potential toward truly automating the process of control parameter tuning for robotic knee prosthesis users.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong