Grasp-TAGS: Task-Aware Grasp Selection via Critic Distillation for Prehensile Loco-Manipulation

Xun Tu, Minghao Zou, Karthik Desingh

November 2026

Simulated success: the robot drags a chair onto the green target pose.

TL;DR Train one RL loco-manipulation policy across several candidate grasps on a chair, cart or bucket, then distill its critic into a small network that picks the grasp before touching the object. In simulation this raises average rearrangement success to 89.84%, from 79.87% for the best single-grasp policy.

What they do

Learned loco-manipulation policies that move large objects such as chairs and carts usually assume one fixed grasp. Grasp-TAGS treats the grasp as a decision. It trains one RL policy over several candidate grasps per object, then picks the grasp before contact by how well that policy is expected to finish the rearrangement. All experiments are in simulation.

How it works

Hierarchical control. A high-level PPO policy outputs base velocities and six arm-joint displacements; a pretrained locomotion policy turns the base commands into 12 leg-joint targets. Trained in Isaac Lab with 4096 parallel environments for 8000 iterations.
Grasp-conditioned training. Each episode samples one of two or three hand-picked grasps (chair armrests or backrest, cart basket sides or handle, bucket edges) and a random planar target pose with yaw within ±90°.
Critic distillation. The PPO critic sees privileged state such as object mass and velocity. Its value at the start of each episode is regressed into a small grasp utility network whose only inputs are the candidate grasp and the goal pose in the object frame. At test time the robot executes the grasp with the highest predicted utility.

Results

Eight objects (three chairs, two carts, three buckets); success means under 0.1 m position error and under 10° keypoint angular error after 750 steps. Average success is 89.84%, versus 79.87% for the best single-grasp policy and 74.07% for the unified policy with a random grasp. Grasp-TAGS is best on every object.
Mean orientation error falls from 0.259 to 0.125 rad against the strongest baseline per object, about 52% lower.
Position error is not uniformly better: 2.60 cm on average, versus 2.12 cm for the best single-grasp policies.

Limitations

Simulation only, with synthetic object models; more objects and real-world deployment on Spot are left to future work. Candidate grasps and their inverse-kinematics solutions are chosen by hand for each object.

Videos

Simulated failure case: the chair ends up misaligned with the green target.

Figures

From Figure 1: For one target pose, the backrest grasp (g1) brings the chair into better alignment with the target th... — From Figure 1: For one target pose, the backrest grasp (g1) brings the chair into better alignment with the target than the two armrest grasps (g2 and g3).

From Figure 2: Pipeline. A hierarchical PPO policy runs on a pretrained low-level controller; its critic is distilled... — From Figure 2: Pipeline. A hierarchical PPO policy runs on a pretrained low-level controller; its critic is distilled into a grasp utility network, which picks the grasp with the highest predicted utility before the policy executes.

From Figure 4: Success rate per object for the best single-grasp policy, the unified policy with a random grasp, and ... — From Figure 4: Success rate per object for the best single-grasp policy, the unified policy with a random grasp, and Grasp-TAGS, over 4096 parallel environments.

Abstract

Loco-manipulation, which requires mobile manipulators to coordinate locomotion and manipulation over long horizons, is a key capability for large-object rearrangement in applications such as service robotics and warehouse automation. Despite recent progress, reliably executing these tasks remains challenging because performance depends on complex robot-object and object-environment interaction dynamics. Existing approaches address this challenge either through model-based methods, which often require accurate dynamics models, or through learning-based methods, which reduce this requirement but typically assume a fixed or predefined initial grasp. However, different geometrically stable grasps can lead to substantially different downstream manipulation performance. Therefore, we propose Grasp-TAGS, a task-aware grasp selection framework for goal-conditioned large-object rearrangement. We train a unified PPO-based reinforcement learning policy across multiple initial grasp configurations and distill the learned critic into a lightweight grasp utility network that ranks candidate grasps using only the candidate grasp configuration and goal information. At inference time, we select the grasp with the highest predicted downstream utility for the downstream policy to complete the task, before even making a contact. Experiments demonstrate that Grasp-TAGS improves rearrangement success over both fixed single-grasp policies and random grasp selection, achieving an average success rate of 89.84% compared with 79.87% for the best-performing single-grasp policies, an improvement of 9.97 percentage points. Grasp-TAGS also reduces final orientation errors across all evaluated objects. For more qualitative results, refer to the supplementary webpage: https://rpm-lab-umn.github.io/Grasp-TAGS-web-page/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong