Leave It Better Than Found: Value-Guided Cooperation in Sequential Tasks

Mario Gomez Andreu, Michael Amir, Antonio Marino, Tianxu An, Marco Hutter, Amanda Prorok

November 2026

TL;DR A follower robot shares its RL value function once, and the leader robot replans online to leave the shared scene in a state the follower values, with no joint training. Follower success rises 27.9 pp on average over five simulated scenarios and from 30% to 70% on real robots.

What they do

When robots from different makers share a space, one robot’s actions change the scene the next one inherits. PAVE (Policy Adaptation via Value Exchange) handles this sequential task continuation without joint training. The follower sends its value function once, and the leader adapts its behavior online so that it finishes its own task and leaves a handoff state the follower values.

How it works

Value as interface. Both robots are trained independently with PPO. The follower’s critic, evaluated at its known start state, scores every shared environment state for the follower.
Planner. The leader runs a receding-horizon Cross-Entropy Method planner, warm-started from its own policy. It maximizes its normalized return plus 4 times the follower’s normalized value of the state at the end of the horizon, using its own value function for rewards beyond the horizon.
OOD filter. The leader may create states the follower never trained on, so the follower’s value is down-weighted by a k-nearest-neighbor distance to 1,000 states from the follower’s training distribution.

Results

Five simulated scenarios (warehouse handoffs, mobile base plus arm, obstacle arena), six seeds: follower success improves by 27.9 pp on average over the naive leader and by up to 48.5 pp (mobile arm: 32.7% to 81.2%), while leader success stays at or above the naive leader’s.
Without the OOD filter, follower success in the blocked-pickup warehouse drops to 32.5%, versus 89.9% with it and 49.8% for the naive leader.
Planning without the follower’s value (leader-only) recovers at most 14.2% of PAVE’s follower gains.
Hardware (two RoboMaster robots, table pushing, 20 trials): follower success rises from 30% to 70%, and the leader succeeds in every trial either way.

Limitations

Only two agents acting in a fixed, known order, and both must share one environment representation. The short planning horizon means the handoff state is approximated by the state at the horizon’s end. The OOD filter is conservative: in the mobile-arm scenario it lowers follower success (81.2% vs 93.8% without it). The leader trusts the follower’s value function, so a malicious follower could steer it into unsafe states. On hardware, floor friction differed from the simulated pushing model, and the leader often needed several push attempts.

Figures

From Figure 1: Naive: the leader pushes the table aside to clean beneath it, unaware of the follower, and leaves it b... — From Figure 1: Naive: the leader pushes the table aside to clean beneath it, unaware of the follower, and leaves it blocking the follower's path.

From Figure 1: PAVE: the follower shares its value function over table positions (heatmap, low to high), and the lead... — From Figure 1: PAVE: the follower shares its value function over table positions (heatmap, low to high), and the leader cleans while pushing the table into a high-value region.

From Figure 1: With the path left clear by the PAVE leader, the follower traverses the room.

From Figure 3: Leader (top) and follower (bottom) return distributions in the five simulated scenarios. Follower retu... — From Figure 3: Leader (top) and follower (bottom) return distributions in the five simulated scenarios. Follower returns are multi-modal because the leader either helps or hinders, and PAVE shifts them toward the success mode.

Abstract

We consider the problem of sequential task continuation, where one robot acts in a workspace that has been affected by a previous robot. Current approaches in multi-robot cooperation rely heavily on co-training or co-optimization, assuming that the robots have access to collective interaction data during the policy training or planning process. However, as increasingly varied robotic embodiments and policies are deployed and operate in the same real-world environment, learning policies for cooperation becomes infeasible due to the combinatorial explosion of possible robot combinations that need to be accounted for. To address this challenge, this paper considers the problem of ad hoc sequential cooperation between robots that have never before operated together in a shared environment. We demonstrate that robots can explicitly communicate their environmental preferences through sharing their value functions, enabling coordination without the need for prior joint training. To ground this exchange and prevent overconfidence in out-of-distribution states, we further introduce a k-nearest neighbors (KNN)-based uncertainty filter. Our method yields an increase of up to 48.5 pp compared with naive or even selfish agents, in task success rates across simulated and real-world experiments, demonstrating that value function transmission between agents is useful not merely as an internal learning tool, but as an effective basis for ad hoc coordination. Website: https://proroklab.github.io/PAVE/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong