GUARD: Anomaly-Informed Q-Steering for Generative Robot Policies in Dexterous Manipulation

Aryaman Gupta, Xiaoyue Wu, Rana Soltani Zarrin, Mark Van der Merwe

November 2026

TL;DR GUARD steers a frozen flow-matching policy at inference time with the gradient of a Q-function trained on a sparse task reward plus a dense anomaly score, so actions stay where the policy is well supported. It reaches 96% average success on four Adroit tasks (21 points above the best baseline) and lifts real Allegro screwdriver rotation from 20% to 60%.

What they do

Diffusion and flow-matching policies cloned from demonstrations drift into states their training data barely covers, and then fail. This is worst in contact-rich dexterous manipulation. GUARD keeps the pretrained policy frozen and nudges its denoising steps with the gradient of a learned Q-function. The Q-function is trained on a sparse success reward plus a dense anomaly score, so it predicts both task return and the risk of leaving the policy’s support.

How it works

Anomaly reward. An anomaly detector (logpZO, a flow model fit to the demonstration observations) scores how poorly a state is supported. Each transition is rewarded for a low score at its next state, standardized and added with equal weight to the standardized sparse task reward. No failure examples or labels are needed.
Q-guided denoising. At each flow step the clean action is estimated in one step, a pessimistic ensemble of five Q-functions is queried there, and its action gradient is added to the flow velocity.
Offline-to-online. The Q-functions are initialized by fitted Q evaluation on noisy rollouts of the base policy, then adapted with rollouts of the steered policy.
Real robot. For the 16-DoF Allegro V5 Plus hand, both stages run entirely in simulation and the result is deployed directly.

Results

Adroit (Relocate, Pen, Hammer, Door): 0.96 average success against 0.60 for the BC policy and 0.75 for DynaGuide, the best baseline. With action-noise perturbations: 0.80 against 0.41 and 0.60.
Sparse reward alone gives 0.66 nominal; guiding directly with anomaly gradients (U-SA) gives 0.72.
GUARD has the lowest average RND anomaly score in both settings.
Real screwdriver tasks, 20 trials each: in-hand rotation goes from 20% to 60% success, with drops falling from 80% to 25%; palm-down reorientation goes from 70% to 95%.

Limitations

On palm-up reorientation GUARD hurts: success falls from 30% to 10%. The authors attribute this to inaccurate contact modeling in simulation, which leaves the Q-function unaware of a failure mode seen on hardware. Q-function training needs online interaction, done here only in simulation. The real-robot policy sees joint positions and object pose; multimodal inputs are left to future work.

Figures

From Figure 1: GUARD steers a pretrained generative policy away from failures. A Q-function learned from sparse task ... — From Figure 1: GUARD steers a pretrained generative policy away from failures. A Q-function learned from sparse task rewards and dense anomaly signals anticipates both task success and the risk of entering anomalous regions, and its gradients guide the denoising process toward successful, well-supported actions.

From Figure 2: Steering in the Adroit Pen task. Left: the unsteered BC policy forms a loose grasp of the pen and fail... — From Figure 2: Steering in the Adroit Pen task. Left: the unsteered BC policy forms a loose grasp of the pen and fails to reorient it, with low Q-values and high state-anomaly scores. Right: when some fingers lose contact, steering applies a larger action correction that folds the fingers back in, keeps a firm grasp and reorients the pen to the target.

From Figure 3: (a) Episode-level failure-detection AUROC on Door using each method's steering signal. (b) Success on ... — From Figure 3: (a) Episode-level failure-detection AUROC on Door using each method's steering signal. (b) Success on Door under increasing action-noise perturbation. (c) Success on Hammer against offline dataset size; dashed is offline-only, solid is offline plus online training. (d) Success on Hammer against online environment steps at the 800-episode offline budget.

From Figure 4: Steering in the real screwdriver in-hand rotation task. Left: the unsteered BC policy drops the screwd... — From Figure 4: Steering in the real screwdriver in-hand rotation task. Left: the unsteered BC policy drops the screwdriver, shown by a sudden drop in Q-value and a jump in anomaly score. Right: whenever the fingers lose hold, steering applies a larger action correction, fixing the grasp twice in the episode and keeping the screwdriver from falling.

Abstract

Generative robot policies based on diffusion and flow matching can model complex, multimodal behaviors, but remain susceptible to covariate shift during closed-loop execution. Existing approaches mitigate such failures through additional demonstrations, policy fine-tuning, or test-time guidance, but often require human intervention, engineered rewards, or other task-specific signals. We introduce GUARD, an anomaly-informed Q-steering framework that leverages the policy's training distribution to construct a dense, failure-anticipatory steering signal. GUARD pairs sparse task rewards with dense anomaly scores learned from policy training data and propagates both through a learned Q-function. Its gradients then steer the denoising process toward actions that improve task return while preventing trajectories from entering poorly supported regions, without directly modifying the pretrained policy. Across four Adroit dexterous manipulation tasks, GUARD achieves average success rates of $96.0\%$ and $80.0\%$ under nominal and perturbed dynamics, outperforming the strongest baseline by $21.0$ and $20.0$ percentage points while maintaining the lowest anomaly score. GUARD further demonstrates direct sim-to-real transfer on in-hand screwdriver rotation and reorientation tasks, achieving $40$ and $25$ percentage-point gains in success rate. These results demonstrate that anomaly signals can serve not only for failure monitoring but also to complement sparse task rewards as effective supervision for robust policy steering.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong