Spectrally Routed Diffusion Policy for Multi-Task Robotic Manipulation
TL;DR SR-DP trains one diffusion policy on several tasks, then adapts it per task by re-weighting the SVD components of its denoiser layers. It averages 68.4% on six RoboTwin tasks, beating per-task specialists by 3.8 points, and raises real bimanual success from 24.0% to 43.0%.
What they do
A diffusion policy trained jointly on several tasks usually does worse than one specialist per task. The authors blame two things: image encoders whose features do not separate tasks that share a workspace, and conflicting gradients from different tasks pushing on the same denoiser weights. SR-DP addresses the first with a better visual condition and the second with a constrained, per-task adaptation of the trained denoiser.
How it works
- Visual condition. A frozen VGGT-1B backbone plus a small adapter pools each view at 1×1, 2×2 and 3×3 into 14 tokens. An auxiliary head predicts the task from these tokens and supplies the task ID at inference.
- Stage 1. A standard DiT diffusion policy is trained on all tasks with no task conditioning.
- Stage 2. Well-conditioned denoiser layers are decomposed by SVD into frozen rank-one experts. A router conditioned on task, token state and diffusion step shifts their singular values, and Householder reflections rotate the input directions. Only 1.2 M parameters (4.1% of the policy head) are trained.
Results
- RoboTwin, six tasks: 68.4% average, versus 64.6% for per-task VO-DP, 46.6% for multi-task VO-DP, 44.5% for SDP and 39.4% for FDP.
- Ablations: the same-size non-spectral adapter reaches 63.3%; dropping the Householder rotations costs 7.5 points.
- Real TeleAvatar dual-arm robot, four bottle tasks × 50 rollouts: Stage 2 averages 43.0%, against 24.0% for Stage 1 and 23.5% for a task-conditioned AdaLN baseline. The largest gains are on refilling the organizer (+26 points) and sorting out the Sprite (+24 points).
Limitations
The SVD basis is only meaningful after Stage 1 converges, so training is not end-to-end. Real-world success stays below 50%, and on the flip task Stage 2 moves only from 13/50 to 15/50, so the authors read the real-world gain only in aggregate.
Figures
Abstract
Diffusion-based policies have achieved strong performance on single-task robotic manipulation. However, a single model trained jointly on multiple tasks consistently underperforms the per-task specialists trained on the same demonstrations. We focus on two challenges. First, the convolutional encoders commonly used in diffusion-based policies produce visual conditions that are not discriminative enough. Second, gradients from incompatible action modes update the shared denoiser's dense parameters along the same directions, causing destructive interference during joint training. To address both, we propose SR-DP (Spectrally Routed Diffusion Policy). A frozen VGGT encoder with a spatial-compression adapter produces compact multi-scale semantic-geometric tokens, a task-discriminative visual condition for tasks that share a workspace. We first train the denoiser as a standard diffusion policy, then decompose selected linear layers by SVD into a frozen basis of orthogonal rank-one experts, adapted through spectral routing and Householder input rotations. On six RoboTwin tasks, SR-DP averages a 68.4% success rate, surpassing multi-task VO-DP by 21.8 points and the strongest per-task baseline by 3.8 points. On four real-world bimanual tasks, Stage 2 reaches a 43.0% average success rate and improves over Stage 1 by 19.0 points.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.