Tactile-RL for Insertion: Generalization to Objects of Unknown Geometry

Siyuan Dong, Devesh K. Jha, Diego Romeres, Sangwoon Kim, Daniel Nikovski, Alberto Rodriguez

Seminal April 2021

TL;DR An RL policy learns on a real robot to correct peg-in-hole misalignment from two GelSlim tactile sensors, using a curriculum of increasingly constrained environments and marker-flow input. Trained on 4 objects, it inserts 4 unseen everyday objects with over 85% average success.

What they do

Peg-in-hole insertion of objects whose geometry is unknown. The robot holds the object between two GelSlim tactile sensors and tries to insert it. When the insertion is blocked, it reads the tactile images from the contact, shifts the object, and tries again. The correction policy is learned with RL entirely on a real robot. The paper ablates the learning scheme, the curriculum, the sensing modality and the tactile representation.

How it works

Episodic RL. TD3 with 12 tactile images per sensor from 0.5 s of contact as the state and a (Δx, Δy, Δθ) gripper-frame correction as the action. The reward combines error reduction, a per-attempt penalty and a success bonus; an episode allows at most 15 attempts. The critic is a small MLP that sees the true contact error instead of images.
Curriculum. Training moves from line wall to corner, U and hole (3 mm clearance, objects about 35 mm wide), with the number of objects growing gradually.
Tactile flow. The input is the marker-displacement field rather than raw RGB, which removes object-specific texture.
Training. The actor is bootstrapped from a supervised policy; training uses about 8 hours of robot time and 3000 data points.

Results

Each policy is tested with 250 insertions per object, with errors within ±5 mm and ±10°.

Full policy: 97-98% on the three rounded training objects and 89.6% on the cuboid (5.42 attempts). Novel objects: small bottle 96.7%, phone charger 96.8%, big bottle 80.8%, paper box 70.1%.
A supervised baseline gets 15.2% on the cuboid. Without the curriculum, the cuboid drops to 57.1% with 11.3 attempts.
RGB input matches on training objects but falls to 37.8% (big bottle) and 13.4% (box).
Force/torque input reaches 99.5% on the big bottle but only 53-61% on the three cuboid-shaped objects, because it struggles with rotation error.

Limitations

Most failures happen when translation and rotation errors are both large: the rotation is too small for the sensors to capture, and the policy can step the wrong way until it diverges. Training needs real robot time, precise grasps and resetting fixtures, because the critic uses the true contact error. Corrections cover only x, y and yaw. The authors note that marker flow may not be the best tactile representation.

Videos

Figures

From Figure 1: Insertion with the tactile RL policy; tactile image sequences from both fingers are the policy input.

From Figure 2: Tactile flow in both sensors for four contact configurations. An edge perpendicular to the sensors rot... — From Figure 2: Tactile flow in both sensors for four contact configurations. An edge perpendicular to the sensors rotates the object in the sensor plane (cases 1 and 2, opposite directions); an edge parallel to the sensors rotates it out of plane (cases 3 and 4).

From Figure 3: Left, the setup: Mitsubishi RV-4FL arm, WSG-32 gripper, two GelSlim sensors, four insertion environmen... — From Figure 3: Left, the setup: Mitsubishi RV-4FL arm, WSG-32 gripper, two GelSlim sensors, four insertion environments, four training objects and four novel everyday objects (phone charger, small bottle, big bottle, box). Right, the TD3 actor: a CNN, a 2-layer LSTM and an MLP over both tactile image sequences.

Abstract

Object insertion is a classic contact-rich manipulation task. The task remains challenging, especially when considering general objects of unknown geometry, which significantly limits the ability to understand the contact configuration between the object and the environment. We study the problem of aligning the object and environment with a tactile-based feedback insertion policy. The insertion process is modeled as an episodic policy that iterates between insertion attempts followed by pose corrections. We explore different mechanisms to learn such a policy based on Reinforcement Learning. The key contribution of this paper is to demonstrate that it is possible to learn a tactile insertion policy that generalizes across different object geometries, and an ablation study of the key design choices for the learning agent: 1) the type of learning scheme: supervised vs. reinforcement learning; 2) the type of learning schedule: unguided vs. curriculum learning; 3) the type of sensing modality: force/torque (F/T) vs. tactile; and 4) the type of tactile representation: tactile RGB vs. tactile flow. We show that the optimal configuration of the learning agent (RL + curriculum + tactile flow) exposed to 4 training objects yields an insertion policy that inserts 4 novel objects with over 85.0% success rate and within 3~4 attempts. Comparisons between F/T and tactile sensing, shows that while an F/T-based policy learns more efficiently, a tactile-based policy provides better generalization.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong