← Paper Digest

LAGR: Language-Aware Grasp Re-Ranking for Task Oriented Grasping

Manav Kulshrestha, Shreyans Jain, Apoorva Vashisth, Ankush Gupta, Damon Conover, Aniket Bera

November 2026

TL;DR LAGR proposes physically feasible grasps from RGB, depth and segmentation, then re-ranks all of them together against a language task instruction. It is trained on a new set of 2.68 million scored grasps, and on a real Franka it executes a task-appropriate grasp in 93 of 100 trials, against 72 for GATER.

What they do

The right grasp depends on the task: a hammer is held by the handle to pound but by the head to hand it over. LAGR separates two questions, whether a grasp is physically feasible and whether it suits the instruction. To train it, the authors build a dataset of 2.68 million scored 5-DoF grasp rectangles over 100,911 image-task pairs from 17 object categories, derived from the HANDAL dataset.

How it works

  • Dataset labels. Candidate grasps are sampled on visible object regions and filtered geometrically. Each gets a feasibility score (a weighted geometric mean of visibility, depth consistency, opening, support, symmetry and clearance) and a utility for transport, functional use or handover; the label is their product.
  • Proposal generator. A MultiMAE encoder fuses RGB, depth and semantic masks, a dense head predicts oriented grasp rectangles with a physical confidence, and the top 96 diverse candidates are kept.
  • Re-ranker. Each proposal’s visual features and geometry are fused with a frozen MiniLM embedding of the instruction, and a transformer scores all proposals together. A small MLP combines this score with physical confidence to pick one grasp.

Results

  • On the benchmark, with the encoder fully fine-tuned, LAGR reaches 85.9% task-conditioned success against 76.6% for GATER, the strongest baseline, with the lowest grasp collision rate (0.84%) at 37.84 ms per sample.
  • Ablations (frozen-encoder default, 82.06%): selecting by physical confidence alone gives 53.03%, and dropping proposal geometry or the proposal-context transformer gives 74.00% and 77.76%.
  • On a real Franka Research 3 (five objects, two tasks each, ten orientations per task): 93/100 successful grasps against 72/100 for GATER. Handovers succeed 92% of the time against 58%, including pan handover at 10/10 against 0/10.

Limitations

Performance depends on the quality and diversity of the generated proposals, and the method covers only planar 5-DoF grasps with a two-finger parallel-jaw gripper. The real tests use a simplified tabletop setup with five objects, and the jug stayed hardest (7/10 for pouring, 6/10 for handover).

Figures

From Figure 1: LAGR picks different grasps on the same real objects for different tasks. Functional tasks favor grasp...
From Figure 1: LAGR picks different grasps on the same real objects for different tasks. Functional tasks favor grasps that enable tool use, while handover tasks leave the handle free for the recipient.
From Figure 3: A task-independent generator fuses RGB, depth and segmentation features and predicts grasp candidates ...
From Figure 3: A task-independent generator fuses RGB, depth and segmentation features and predicts grasp candidates with physical confidence. A language-conditioned re-ranker scores them jointly with the instruction and selects one without changing its geometry.
From Figure 2: Example grasp annotations from the LAGR dataset (top row) next to those of existing datasets.
From Figure 2: Example grasp annotations from the LAGR dataset (top row) next to those of existing datasets.
Abstract

Grasping is a fundamental capability for robotic manipulation, with the appropriate grasp depending on the downstream task. For example, a hammer should be held by its handle for hammering but by its head when it is handed over. Selecting such a grasp requires consideration of two criteria: physical feasibility on the object and suitability for the task. Developing such an approach requires dense grasp labels with both feasibility and task scores on real observations, but existing task-oriented grasping datasets are small, synthetic, or costly to annotate. To address this gap, we construct a dataset of 2.68 million scored 5-DoF grasp annotations over 100,911 image-task pairs from $17$ object categories. Additionally, we introduce LAGR, a task-oriented grasp detection method that explicitly addresses both these challenges. Given vision-language inputs, LAGR first generates task-independent grasp proposals along with a physical feasibility score, and then scores the generated grasps with a language-conditioned re-ranker that reasons over all proposals jointly. Experiments show that LAGR outperforms the next best method by 9.3 percentage points. We further validate LAGR on real objects by executing task-oriented grasps with a Franka Research 3 robot.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.