Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks

Roberto Martín-Martín, Michelle A. Lee, Rachel Gardner, Silvio Savarese, Jeannette Bohg, Animesh Garg

Seminal June 2019

TL;DR An RL policy outputs both end-effector pose targets and the stiffness of an impedance controller (VICES), compared against joint-space and fixed-impedance action spaces on path following, door opening and wiping. VICES earns the highest wiping reward at a 28 N mean force, transfers from a simulated Panda to a Sawyer, and wipes a real whiteboard in 8 of 10 trials.

What they do

An empirical study of how the action space affects RL for manipulation. It compares joint torques, velocities and positions, variable joint impedance, and end-effector impedance with low, medium or high fixed gains, and advocates variable impedance control in end-effector space (VICES), where the policy also outputs the controller’s stiffness and damping along all six axes.

How it works

Controller as action interface. The policy runs at 20 Hz. An analytic controller turns each action into joint torques at 500 Hz, interpolating linearly between consecutive actions.
VICES. The policy outputs a desired end-effector pose change plus position and orientation gains. An operational-space impedance controller maps these to end-effector wrenches and then to joint torques through the Jacobian.
Tasks. Free-space path following through four via-points, opening a door to 60°, and wiping a table at an unknown height from 48x48 RGB images. All are trained with PPO on a simulated Panda, five seeds per action space.

Results

Path following: every action space except joint torques and low fixed impedance passes all four via-points. Among end-effector policies, only VICES consistently cut energy use without losing task performance once an energy penalty was added.
Door opening: medium, high and variable end-effector impedance reach close to 100% success. Joint velocity reaches up to 75%, while joint torques and positions fail.
Wiping: VICES converges faster, reaches the highest reward and wipes the largest fraction of the stain, at a mean force of 28 N (below the 40 N payload).
Zero-shot transfer: in simulation, only end-effector policies transfer from Panda to Sawyer. On the real robot, without force sensing, the VICES policy wipes a whiteboard in 8 of 10 trials.

Limitations

Only PPO was tested. Transferred end-effector policies still lose performance: they do not account for the new robot’s joint limits, and when wiping, the other robot produces different forces. For sim-to-real, camera images are replaced by simulator renders with a color-segmented overlay.

Figures

From Figure 1: A policy learned with variable impedance control in end-effector space (VICES) on one simulated robot ... — From Figure 1: A policy learned with variable impedance control in end-effector space (VICES) on one simulated robot transfers to another simulated robot and to the real robot, because the controller compensates for each robot's kinematics and dynamics.

From Figure 2: The policy outputs actions at 20 Hz. Depending on the action space (joint torques, joint velocity, joi... — From Figure 2: The policy outputs actions at 20 Hz. Depending on the action space (joint torques, joint velocity, joint position or end-effector pose), a controller combines the target, the gains (fixed or given by the policy) and the robot state into joint torques at 500 Hz, interpolating linearly between policy actions.

From Figure 3: Surface wiping training curves, mean and standard deviation over five seeds. The policy using variable... — From Figure 3: Surface wiping training curves, mean and standard deviation over five seeds. The policy using variable impedance in end-effector space (blue) reaches the highest reward because it learns to adapt how much force it applies.

From Figure 6: The VICES policy trained in simulation runs directly on the real robot. It presses compliantly toward ... — From Figure 6: The VICES policy trained in simulation runs directly on the real robot. It presses compliantly toward the table (1); when the experimenter marks new parts of the whiteboard (2, 4), it moves there (3). Plots show height above the table and the forces in x, y and z.

Abstract

Reinforcement Learning (RL) of contact-rich manipulation tasks has yielded impressive results in recent years. While many studies in RL focus on varying the observation space or reward model, few efforts focused on the choice of action space (e.g. joint or end-effector space, position, velocity, etc.). However, studies in robot motion control indicate that choosing an action space that conforms to the characteristics of the task can simplify exploration and improve robustness to disturbances. This paper studies the effect of different action spaces in deep RL and advocates for Variable Impedance Control in End-effector Space (VICES) as an advantageous action space for constrained and contact-rich tasks. We evaluate multiple action spaces on three prototypical manipulation tasks: Path Following (task with no contact), Door Opening (task with kinematic constraints), and Surface Wiping (task with continuous contact). We show that VICES improves sample efficiency, maintains low energy consumption, and ensures safety across all three experimental setups. Further, RL policies learned with VICES can transfer across different robot models in simulation, and from simulation to real for the same robot. Further information is available at https://stanfordvl.github.io/vices.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong