LAGR: Language-Aware Grasp Re-Ranking for Task Oriented Grasping

TL;DR LAGR proposes physically feasible grasps from RGB, depth and segmentation, then re-ranks all of them together against a language task instruction. It is trained on a new set of 2.68 million scored grasps, and on a real Franka it executes a task-appropriate grasp in 93 of 100 trials, against 72 for GATER.
What they do
The right grasp depends on the task: a hammer is held by the handle to pound but by the head to hand it over. LAGR separates two questions, whether a grasp is physically feasible and whether it suits the instruction. To train it, the authors build a dataset of 2.68 million scored 5-DoF grasp rectangles over 100,911 image-task pairs from 17 object categories, derived from the HANDAL dataset.
How it works
- Dataset labels. Candidate grasps are sampled on visible object regions and filtered geometrically. Each gets a feasibility score (a weighted geometric mean of visibility, depth consistency, opening, support, symmetry and clearance) and a utility for transport, functional use or handover; the label is their product.
- Proposal generator. A MultiMAE encoder fuses RGB, depth and semantic masks, a dense head predicts oriented grasp rectangles with a physical confidence, and the top 96 diverse candidates are kept.
- Re-ranker. Each proposal’s visual features and geometry are fused with a frozen MiniLM embedding of the instruction, and a transformer scores all proposals together. A small MLP combines this score with physical confidence to pick one grasp.
Results
- On the benchmark, with the encoder fully fine-tuned, LAGR reaches 85.9% task-conditioned success against 76.6% for GATER, the strongest baseline, with the lowest grasp collision rate (0.84%) at 37.84 ms per sample.
- Ablations (frozen-encoder default, 82.06%): selecting by physical confidence alone gives 53.03%, and dropping proposal geometry or the proposal-context transformer gives 74.00% and 77.76%.
- On a real Franka Research 3 (five objects, two tasks each, ten orientations per task): 93/100 successful grasps against 72/100 for GATER. Handovers succeed 92% of the time against 58%, including pan handover at 10/10 against 0/10.
Limitations
Performance depends on the quality and diversity of the generated proposals, and the method covers only planar 5-DoF grasps with a two-finger parallel-jaw gripper. The real tests use a simplified tabletop setup with five objects, and the jug stayed hardest (7/10 for pouring, 6/10 for handover).
Figures



Abstract
Grasping is a fundamental capability for robotic manipulation, with the appropriate grasp depending on the downstream task. For example, a hammer should be held by its handle for hammering but by its head when it is handed over. Selecting such a grasp requires consideration of two criteria: physical feasibility on the object and suitability for the task. Developing such an approach requires dense grasp labels with both feasibility and task scores on real observations, but existing task-oriented grasping datasets are small, synthetic, or costly to annotate. To address this gap, we construct a dataset of 2.68 million scored 5-DoF grasp annotations over 100,911 image-task pairs from $17$ object categories. Additionally, we introduce LAGR, a task-oriented grasp detection method that explicitly addresses both these challenges. Given vision-language inputs, LAGR first generates task-independent grasp proposals along with a physical feasibility score, and then scores the generated grasps with a language-conditioned re-ranker that reasons over all proposals jointly. Experiments show that LAGR outperforms the next best method by 9.3 percentage points. We further validate LAGR on real objects by executing task-oriented grasps with a Franka Research 3 robot.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
