Deploying Physically and Epistemically Compliant Agents by Distilling Belief-Space Task and Motion Planners

Phillip J. K. Christoffersen, Rachel Ma, Patrick Barragán, Dylan Hadfield-Menell

November 2026

TL;DR Plan offline with a belief-space task-and-motion planner, then distill the plans into a history-based policy that acts on raw images and language. In the new Scavenger search domain the policy succeeds over 90% of the time, against at most 5% for reward-engineered PPO.

What they do

Some rules depend on what the agent knows, for example “do not take an irreversible action before you have the needed information”. RL rewards for such rules are hard to optimize, and online belief-space planners need hand-built belief estimators. This paper runs a belief-space task-and-motion planner offline in a simulator and distills its plans into a policy that sees only raw observations. Everything is evaluated in a 2D simulated domain.

How it works

Planner. Fast Downward picks tasks and sequential quadratic programming solves motions. The belief is a particle filter; the planner assumes the observation of the most likely state and replans when execution disagrees.
Hierarchical policy. An LSTM high-level policy predicts the current planner action from the observation-action history, and MLP low-level policies output controls.
Dual Averaging. The high-level loss adds the average over timesteps where the task changes to the average over all timesteps, countering the bias toward continuing the current action.
ρ-Optimal Reinforcement. A successful rollout no longer than ρ = 1.1 times the planner’s trajectory is added to the training data; otherwise the planner’s trajectory is.

Results

Scavenger: find a hint in a 3-6 room floorplan, then go to the right one of three markers and guess, without hitting walls or guessing early. Over 90% success with visual and with language hints; flat and hierarchical PPO stay at or below 5% after 24 hours or more with dense reward.
Held-out commands cause virtually no degradation, and intentionally vague commands are followed more than 60% of the time.
Larger floorplans: 80% success with 8 rooms, 50% with 12 and 36% with 18, three times the training maximum.
Theory: in static search with noiseless observations, a planner that covers at least δ of the prior per replan becomes certain within O(1/δ) replans.

Limitations

The method assumes a tractable belief-space planning model of the problem can be written. The completeness result covers only static search with noiseless observations. Experiments are in one simulated domain with simple position control; there is no real robot.

Figures

From Figure 1: A worked example, where the robot must obey the epistemic rule “ensure the user leaves their home, and... — From Figure 1: A worked example, where the robot must obey the epistemic rule “ensure the user leaves their home, and has what they need when they leave” while being physically safe. Left: the robot first searches the house for the user, who may be in one of three locations. Middle: after finding the user, the robot asks what they need for the day. Right: the robot hears the user plans a drive on a rainy day, and brings their keys and umbrella.

From Figure 2: A sample initial condition in Scavenger, a procedurally generated multiroom floorplan with circle, squ... — From Figure 2: A sample initial condition in Scavenger, a procedurally generated multiroom floorplan with circle, square and triangle floor markers and markers for possible hint locations.

From Figure 3: Success rate against wall-clock training time in Scavenger for the method, its three ablations (no pol... — From Figure 3: Success rate against wall-clock training time in Scavenger for the method, its three ablations (no policy feedback, no ratio cap, no task-change term), task exploration node feedback, hierarchical PPO and flat PPO.

From Figure 4: Results of the generalization study. The policy trained with language hints transfers to unseen langua... — From Figure 4: Results of the generalization study. The policy trained with language hints transfers to unseen language commands, to intentionally vague commands, and to problems with more rooms than seen in training. Each success rate is calculated from 50 episodes, and the error bars are binomial standard errors.

Abstract

In this work, we train agents to act under uncertainty according to specified, belief-dependent rules. Deep reinforcement learning (RL) can in theory train agents to follow rules if encoded into well-designed rewards, but in practice, RL fails to perfectly optimize these rewards, leading to ineffective or noncompliant agent behavior. Belief-space planners by construction produce behavior in line with developer specification, but require hand-programming online planners and belief-state estimators, adding significant engineering complexity at scale. We combine the two: we plan offline to generate a dataset of sample trajectories and then distill the plans into a policy that acts on raw observations. We propose two changes to planner distillation that address problems specific to history-based policies: Dual Averaging, a loss that combines per-timestep predictive loss and average loss on task changes, and $ρ$-Optimal Reinforcement, a feedback mechanism that reinforces the policy's own successful rollouts when these are near-optimal. We provide a sufficient condition under which static search problems with noiseless observation models is complete, and show that maximum-likelihood-observation planning implies this condition in static search with continuous beliefs. We create a challenging new domain, Scavenger, where an agent needs to gain information in order to succeed. We evaluate our distilled policy in two versions of Scavenger, one where information is provided visually and one where it is provided as a command in natural language. In either case, the learned policy attains a success rate above 90% in Scavenger, compared with at most 5% for RL. We also find that distilled planners transfer with greater than 60% success to following unseen and intentionally vague commands and greater than 75% success to following floorplans larger than those seen in training.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong