Cleanly-DINO: Efficiently Learning Visual Task Spaces for Robotics

TL;DR Fits a small projection on frozen DINOv2 features from a few cleaning videos, so that every image patch gets a dirty-to-clean progress score. It separates clean from dirty patches with 0.763 AUROC on held-out videos. A real robot that wipes the least clean patch first was rated at about 83% of human cleaning quality.
What they do
In tasks like cleaning, progress shows up as a change in appearance, not in an object’s pose. Cleanly-DINO learns a per-patch progress score from a handful of demonstration videos and a mask of the task region, with no timestamps, goals or language. The authors test it on a hand-labeled dataset of six cleaning tasks, as an observation space and reward for RL, and as the guide for a real wiping robot.
How it works
- Frozen DINOv2 (ViT-S) embeds every frame. Patches whose features change strongly over a demonstration are labeled as going from dirty to clean, and the rest count as clean.
- Trajectory-aligned abstention drops changing patches that do not move along the demonstration’s main direction of change. In a ViT, a patch’s features also shift when nearby patches change.
- A contrastive loss trains a projection to 8 dimensions that pushes dirty and clean patches apart. A patch’s progress is its position between the mean start and end anchors.
- On the robot, a wiping primitive presses with about 10 N and makes a 10 cm stroke from the least clean patch. It repeats until every patch passes a threshold.
Results
- The dataset has 53 demonstrations, split into 28 for training and 25 held out. With abstention, held-out within-frame AUROC is 0.763 (linear head) and 0.771 (MLP head). Raw DINOv2 reaches 0.721, and TCN, PROGRESSOR and TimeRewarder reach at most 0.655.
- A single in-domain demonstration helps more than any amount of data from other tasks. Adding other tasks helps up to three of them and barely hurts tasks already learned.
- As the observation space for simulated RL wiping, it loses about half as much success as raw DINOv2 under visual changes. Under a dirt color shift, for example, it loses 1.7 points against DINOv2’s 22.5, from about 81%. As a reward, it does no better than TCN or TimeRewarder.
- On the real robot, the tests covered two tablecloths and a wooden table, three condiments, and three runs each. 60 survey responses rated the robot 3.64 ± 1.09 and human wiping 4.39 ± 0.94 out of 5.
Limitations
Transfer is limited, since one in-domain demonstration beats all cross-domain data. On a dark floral tablecloth the model marked the whole table as dirty. The final progress readout varies by scene, so the robot stopped at a fixed threshold and action limit. Task masks were made by hand with SAM2.
Figures




Abstract
Learning manipulation tasks from few demonstrations requires abstractions that capture the changes relevant to task completion, such as object poses in rigid-body manipulation. However, tasks such as cleaning are defined by changes in visual state rather than object pose, making the choice of task representation less clear. In this work, we propose Cleanly-DINO to learn visual task spaces from few demonstrations. Given a mask of a task-relevant region and frozen ViT features, Cleanly-DINO fits a projection to isolate a task space and fit a progression metric to it. We train the model on diverse cleaning examples and evaluate its ability to localize incomplete task regions and transfer across visual domains. We also study its utility for robot learning by using it as an observation encoder and reward signal in a reinforcement learning setting. Finally, we deploy our fitted model within a real-world robotic wiping pipeline to evaluate its use in guiding task execution.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
