Grana: Compositional Reward Modeling for Long-Horizon Robot Manipulation
TL;DR A vision-language reward model picks the unfinished subtask from a language list, predicts its progress and adds credit for the finished subtasks. Pretrained on 200 tasks, it cuts zero-shot progress error by 20.7% over RoboMeter, and progress-weighted cloning raises real airplane-assembly success from 50% to 75%.
What they do
Grana is a progress-based reward model for multi-minute assembly tasks, where a failure such as dropping a screw should reset the current subtask without erasing credit for the finished ones. A vision-language model selects the unfinished subtask from an ordered list of language descriptions and predicts that subtask’s local progress p. With k subtasks finished out of K, whole-task progress is (k + p)/K. The model is pretrained on 200 mostly short tasks and then adapted to long ones.
How it works
- Shared predictor. Qwen3.5-4B is called twice: once to select the subtask, once to predict progress for it. Each call sees eight timesteps from four cameras. LoRA adapters, a temporal transformer and a binned progress head are trained; the base model stays frozen.
- Structured supervision. Annotators mark subtask boundaries and intervals of suboptimal or regressed execution, not numeric scores. Nominal frames advance progress 20x faster than suboptimal ones. A regression resets local progress to zero, while completed subtasks keep their credit.
- Order-aware targets. When the model queries the wrong subtask, earlier subtasks get a target of 1 and later ones 0.
- Policy use. Progress increments become rewards for offline RL (IQL, then BPPO), or per-step weights (1, 0.5, 0) for behavior cloning of a GR00T-2 world-action model.
Results
- On 50 held-out tasks, zero-shot RMSE is 0.180 vs 0.227 for RoboMeter. Going from 15 to 200 pretraining tasks cuts RMSE by 35.3%.
- On five long tasks (up to 18 subtasks), each adapted with 15 labeled trajectories, Grana has the lowest RMSE on every task, 16.1-31.6% below the best baseline. Moving from the YAM robot to the Tianji robot: 0.452 vs 0.556.
- Offline RL with zero-shot rewards reaches 65.0% on star insertion and 66.7% on tube transfer, vs 50.0% and 41.7% for the best other reward source.
- In the second round of weighted cloning, success is 75% vs 50% on airplane and 67% vs 42% on car assembly. Using Grana to trigger and guide MPC raises tractor assembly from 2/10 to 9/10.
Limitations
Not discussed in a section of their own. Composing progress assumes completed subtasks stay completed, and a wrong subtask selection still distorts the estimate. Because the model can use later frames, the policy experiments score recorded trajectories offline. Long tasks need annotated adaptation trajectories, and each policy result rests on 12 trials.
Videos
Figures




Abstract
Robot reward modeling faces two challenges: generalizing across tasks and accurately tracking progress in long-horizon manipulation. General-purpose reward models demonstrate broad task transfer, but transfer from short-task pretraining to long-horizon tasks remains underexplored. Long-horizon progress models often rely on task-specific subtask classifiers or fixed action vocabularies, limiting the reuse of learned progress knowledge when adapting to new tasks. To address these challenges, we introduce Grana, a vision-language robot reward model that learns to identify unfinished subtasks and estimate their continuous progress using open-vocabulary subtask descriptions. We train a language-conditioned model to estimate progress in short-horizon tasks and individual subtasks of long-horizon tasks. For long-horizon tasks, we combine the selected subtask's predicted progress with credit for preceding completed subtasks to estimate whole-task progress. General progress prediction requires training the model across diverse tasks with progress supervision that accounts for imperfect execution. Elapsed-time proxies can misrepresent progress when relative subtask durations vary, or failures reverse local progress. We therefore introduce an annotation protocol that converts sparse subtask boundaries and action-quality annotations into dense progress targets, without requiring annotators to assign numerical progress scores. Pretraining with this supervision on 200 diverse, mostly short-horizon manipulation tasks enables zero-shot reward prediction on unseen tasks and improves progress prediction after adaptation to multi-minute, long-horizon tasks. Experiments demonstrate (i) generalizable reward prediction, including adaptation to five long-horizon tasks, (ii) benefits from pretraining and composing subtask progress estimates, and (iii) improved downstream policy performance.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
