Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervision

Kuan Fang, Yuke Zhu, Animesh Garg, Andrey Kurenkov, Viraj Mehta, Li Fei-Fei, Silvio Savarese

Seminal June 2018

TL;DR TOG-Net learns where to grasp a tool and how to swing it together, trained only on simulated trial and error with procedurally generated tools. On a real Sawyer with 9 unseen objects it reaches 71.1% success at sweeping and 80.0% at hammering.

What they do

A grasp that is stable is not always a grasp that lets the robot use the tool. Dex-Net-style grasping picks the most robust grasp, often near the center of mass, but hammering wants the end of the handle and sweeping wants the head. TOG-Net jointly learns a task-oriented grasp and the manipulation action that follows it, trained only from simulated trial and error.

How it works

Factored grasp score. Task success probability is split into grasp quality (will the lift succeed?) times task quality given a successful grasp. 200 antipodal candidates from a depth image are scored by this product and refined with 3 iterations of the cross-entropy method.
Manipulation policy. A Gaussian policy predicts the start pose (x, y, z, rotation) of a fixed open-loop motion primitive: a 40 cm straight sweep or a 90-degree wrist swing for hammering. It is trained with policy gradient, using task success as the reward.
Self-supervision. One network with shared convolutional layers predicts all three outputs. It is trained in Bullet on 18,000 procedurally generated two-part tools (T-, L- and X-shapes), over three rounds of 500K trials with epsilon-greedy exploration.

Results

In simulation (100,000 episodes per method) the full model beats all four baselines on primitive, complex and realistic test objects for both tasks.
Real world, 9 unseen objects, 270 trials: sweeping 71.1% vs 24.4% (antipodal + trained policy) and 23.6% (task-agnostic + trained policy); hammering 80.0% vs 33.3% and 44.4%.
Miscellaneous objects unlike the training shapes are hardest: 60.0% sweeping and 66.7% hammering.

Limitations

Grasps are restricted to top-down, 4-DoF, and manipulation is a single open-loop motion primitive per task. Training tools are two convex parts joined together, and the model sometimes misreads novel parts, e.g. sweeping round cans with a pan’s round body. With a parallel-jaw gripper it grasps closer to the hammer head than ideal, since a far grasp is hard to balance.

Videos

Figures

From Figure 1: The same hammer grasped three ways. A task-agnostic grasp lifts it, but sweeping and hammering each ne... — From Figure 1: The same hammer grasped three ways. A task-agnostic grasp lifts it, but sweeping and hammering each need a different grasp.

From Figure 2: Model overview. Grasp candidates are scored by a task-oriented grasping model, and a manipulation poli... — From Figure 2: Model overview. Grasp candidates are scored by a task-oriented grasping model, and a manipulation policy outputs actions conditioned on the chosen grasp. Both are trained jointly with simulated self-supervision.

From Figure 5: Simulated sweeping success on primitive, complex and realistic objects. The full model beats all four ... — From Figure 5: Simulated sweeping success on primitive, complex and realistic objects. The full model beats all four baselines in every category.

From Figure 5: Simulated hammering success on the same object categories. The gap to the baselines is larger than for... — From Figure 5: Simulated hammering success on the same object categories. The gap to the baselines is larger than for sweeping.

Abstract

Tool manipulation is vital for facilitating robots to complete challenging task goals. It requires reasoning about the desired effect of the task and thus properly grasping and manipulating the tool to achieve the task. Task-agnostic grasping optimizes for grasp robustness while ignoring crucial task-specific constraints. In this paper, we propose the Task-Oriented Grasping Network (TOG-Net) to jointly optimize both task-oriented grasping of a tool and the manipulation policy for that tool. The training process of the model is based on large-scale simulated self-supervision with procedurally generated tool objects. We perform both simulated and real-world experiments on two tool-based manipulation tasks: sweeping and hammering. Our model achieves overall 71.1% task success rate for sweeping and 80.0% task success rate for hammering. Supplementary material is available at: bit.ly/task-oriented-grasp

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong