All-For-One: Distilling Visual Foundation Models to a Fast, Memory-Efficient Visuomotor Policy

Byungkun Lee, Minho Park, Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Kyungmin Lee, Yunho Kim, Jaegul Choo

November 2026

TL;DR AFO trains a teacher policy on RGB plus SAM 2 masks and Depth Anything V2 depth, then distills its features into a 2.7M-parameter RGB-only student that runs no foundation model at inference. The student averages 92.57% on seven Adroit and MetaWorld tasks (84.57% when trained from scratch at the same 2.11 ms latency), and on a real Indy7 arm it reaches 52.59% under visual shifts, against 43.33% for ACT.

What they do

Vision foundation models (VFMs) give depth and segmentation that help manipulation, but running them in the control loop is slow, and their per-frame outputs can be wrong or flicker. All-For-One uses them only during training: a teacher policy sees their outputs, and a small RGB-only student learns to reproduce the teacher’s features.

How it works

All (teacher). Molmo points to the target object in the first frame from the task description, SAM 2 segments and tracks it, and Depth Anything V2 predicts depth for every frame. Separate learnable encoders embed RGB, mask and depth; the embeddings are summed, joined with proprioception and trained with behavior cloning.
For-One (student). A single RGB encoder with the same policy head, trained with behavior cloning plus a cosine loss that aligns its embedding with each of the teacher’s three modality embeddings. The distillation weight decays to zero as the student’s success rate catches up with the teacher’s.
Training uses only downstream robot demonstrations (25-100 per simulated task, 35 per real task). The real-robot policies use an action-chunking transformer head with ResNet-18 encoders.

Results

Simulation (7 tasks, 5 seeds): the student averages 92.57%, against 84.57% for the same network trained from scratch, 89.14% for R3M, 87.43% for Theia and 87.09% for its own teacher. The student has 2.7M parameters and takes 2.11 ms per step; the teacher has 8.2B and takes 110.07 ms.
Real Indy7 arm, 3 tasks scored by task progress: 64.52% in distribution (ACT 58.80%, Theia 58.53%), and 52.59% under desk-mat, lighting, clutter and placement shifts (43.33% and 43.14%).
On Adroit Relocate the teacher succeeds in 87% of rollouts without a VFM error and 45% with one. A teacher given ground-truth masks and depth averages 94.40% over the seven simulated tasks.

Limitations

Only lightweight visual encoders are studied, not large VLA backbones, which may cap absolute performance and generalization. That the student beats its teacher because it avoids VFM errors is, in the authors’ words, a conjecture. MetaWorld is near ceiling for every method, and only three of seven comparisons there are statistically significant.

Figures

From Figure 1: Overcoming the compute-performance tradeoff. Learning from scratch (LfS) is fast but inaccurate, where... — From Figure 1: Overcoming the compute-performance tradeoff. Learning from scratch (LfS) is fast but inaccurate, whereas VFM backbones are accurate but slow. By distilling large pretrained VFMs into a compact student, AFO attains high success rates while staying efficient at inference.

From Figure 2: Overview of All-For-One. (a) All: a teacher uses Molmo, SAM 2 and Depth Anything V2 with learnable lig... — From Figure 2: Overview of All-For-One. (a) All: a teacher uses Molmo, SAM 2 and Depth Anything V2 with learnable lightweight encoders for RGB, object mask and depth; their representations are summed and train a policy with a BC loss. (b) For One: a lightweight RGB-only student encoder learns from the teacher through a feature distillation loss plus the BC loss.

From Figure 3: Benchmarks. Adroit and MetaWorld tasks in simulation, and clutter sweep, clean toolbox and pick-and-pl... — From Figure 3: Benchmarks. Adroit and MetaWorld tasks in simulation, and clutter sweep, clean toolbox and pick-and-place on the real robot, both in distribution and under visual shifts such as desk mat and lighting changes and added clutter.

From Figure 4: Failure modes of the VFM-based teacher. (a) SAM 2 misses the target or masks an irrelevant object. (b)... — From Figure 4: Failure modes of the VFM-based teacher. (a) SAM 2 misses the target or masks an irrelevant object. (b) Depth Anything V2 predicts a shadow as a closer surface. (c) Depth estimates fluctuate across adjacent frames. (d) On Adroit Relocate the teacher succeeds 87% of the time without VFM errors, but only 45% when an error occurs.

Abstract

Visual Foundation Models (VFMs) provide semantic cues such as depth and segmentation useful for robotic manipulation, but directly deploying them in closed-loop control is impractical due to high computational cost, temporal inconsistency, and distribution shift between static internet images and egocentric robot observations. We propose All-For-One (AFO), a two-stage distillation framework that leverages VFMs strictly as privileged information during training, using downstream robotic data. Following the privileged learning paradigm, AFO decouples semantic representation learning from inference-time policy execution. In the All stage, a teacher policy fuses RGB observations with outputs from multiple VFMs to learn a semantically enriched representation. In the For-One stage, this knowledge is distilled in-distribution into a lightweight RGB-only student policy. Evaluated on two Adroit tasks and five MetaWorld tasks in simulation, as well as three real-world manipulation tasks, AFO outperforms prior foundation-model-based approaches while using over 30x fewer parameters than VFM-based encoders and 50x lower latency than its own teacher in simulation benchmarks, providing an efficient pathway for transferring foundation-model knowledge to robotic control.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong