World In-Context Learning for Agentic Robot Manipulation
TL;DR A frozen foundation model learns each manipulation task from one demonstration by rehearsing in a simulator and writing what it learns into a text Task Memory instead of weights. RoboScript reaches 83.4% mean success on the 50 RoboTwin 2.0 tasks, against 61.7% for the best trained policy, and the memories transfer to a real dual-arm robot.
What they do
Trained manipulation policies store skill in weights and lose it under distribution shift. In-context learning from one demonstration avoids retraining, but the demonstration carries no feedback from the scene in front of the robot. World In-Context Learning (WICL) adds that feedback: a frozen foundation model starts from one demonstration, rehearses the task in a simulator, and writes what it learns into its context. RoboScript implements this for dual-arm manipulation.
How it works
- Tools with receipts. The agent sees RGB-D and proprioception only, and acts through grasp, place and motion entry points plus skills it writes itself. Each action returns a receipt (commanded and achieved pose, gripper opening, whether an object is held).
- Task Memory. A Markdown document of object roles, workflow, relative grasp and place specifications and lessons. A linter rejects coordinates, ground-truth terms and seed identifiers.
- Learning. The agent imitates the single demonstration, then rehearses on new clean scenes; a revision is merged only if it makes the failed instance succeed. The memory is then frozen.
- Deployment and reflection. The agent chooses the arm per scene, recovers from failed actions without reset and checks the final state against the demonstration. A reflection agent reads episode records and writes guidance for the next block of unseen seeds.
Results
- RoboTwin 2.0, 50 tasks, 10 episodes per task and setting: 83.6% clean and 83.2% randomized with GPT-6.1-Sol. The best trained policy, 4D-WAM with 50 demonstrations per task, gets 81.5% and 41.8%.
- Ablations on 10 tasks: without rehearsal the demonstration alone gives 69.0% vs. 78.5%; rehearsal without a demonstration gives 56.5% vs. 86.0%; removing recovery gives 69.0%.
- Reflection nets +18, +7 and +1 episodes over three rounds on all 50 tasks.
- Real dual-arm PiPER, memories remapped by an adapter, actions planned offline: 10/10, 7/10, 7/10 and 6/10 on four tasks.
Limitations
Depends on commercial model APIs whose quality shifts success by several points between sessions. Inference is slow: 54.7 model calls per episode, and learning plus evaluation cost 1,911 USD with GPT-6.1-Sol. Reflection gains shrink quickly. Only rigid and articulated objects were tested; deformables and tightly coordinated bimanual contact such as handovers remain hard.
Figures
Abstract
Trained action policies write manipulation skill into network weights, tying what is learned to the distribution it is learned from. In-context learning (ICL) avoids this coupling but has not matched trained policies, because a demonstration recorded in another scene carries no feedback from the world the robot currently faces. We observe that what an agent learns in context is semantic, recording task steps and object roles rather than visual appearance, so experience gathered in one world transfers to another. Based on this insight, we propose World In-Context Learning (WICL), a paradigm in which a frozen model starts from a demonstration, rehearses the task in a world such as a simulator, and writes the feedback into its context rather than its weights. We build RoboScript, a modular infrastructure that implements WICL for agentic manipulation. From a single demonstration, RoboScript learns each task through rehearsal in simulation, freezes the learned knowledge as a Task Memory, and deploys the frozen memory on new scenes with data isolation that ensures every deployment input is available on a real robot. On all 50 tasks of RoboTwin 2.0, RoboScript reaches 83.4% mean success across clean and randomized settings with one demonstration per task and no gradient updates, exceeding the strongest trained action policy by over 20 points. The method generalizes across visual distributions, improves through reflection, scales with the foundation model, and transfers to a physical dual-arm robot through cross-embodiment memory adaptation. With a world in the loop, a robot can learn a manipulation task in its context as well as it learns one in its weights. All code is available at https://github.com/ArcheOne-AI/RoboScript
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.