Objectives as Policies: Vision-Language Model Cost Programs for Contact-Rich Manipulation
TL;DR A VLM writes a multi-stage cost program for a sampling-based MPC from a fixed library of cost terms, refines it on rollouts in a real-to-sim digital twin, and the same program then runs on the real arm. Across four contact-rich tabletop tasks with moved objects, programs that succeed in simulation succeed in 32 of 40 real executions.
What they do
Contact-implicit MPC handles contact well but needs costs engineered for each task and scene; vision-language-action models bring semantics but no physics model. Objectives as Policies (OaP) has a VLM write the MPC’s objective as an editable program, tests it in a digital twin of the scene, revises it, and then runs it on a Flexiv Rizon 4s arm with a parallel gripper. No task-specific training.
How it works
- Objective program. A sequence of stages, each with running costs, terminal costs, weights and completion thresholds, built from a fixed library of residuals: distances, axis alignment, rotation progress, region membership, tool-mediated contact, object held, gripper command. A stage verifier advances once every terminal residual is under its threshold.
- Real-to-sim. Meshes from SAM 3D, scale and pose refined with Any6D, density and friction estimated by a VLM; FoundationPose tracks objects on hardware.
- MPC. MPPI with 1000 rollouts over 600 physics steps in MuJoCo-warp, replanning every 40 controls.
- Refinement. GPT-5.6-sol drafts and self-reviews the program, then revises it from six subsampled state records of each failed simulated rollout.
Results
- Simulation (four tasks, 10 trials each): self-review raises first-try success from 20/40 to 26/40, and up to five feedback rounds reach 39/40.
- With two feedback rounds OaP succeeds in 36/40, against 31/40 for an adapted CoRAL, 17/40 for ReKep, 11/40 for Language to Rewards, 10/40 for pi0.5 and 0/40 for OpenVLA-OFT (released checkpoints, no fine-tuning).
- Without the cost library, first-execution success with self-review falls from 65% to 35%.
- Real robot: 32/40 executions with object positions perturbed. Most failures were poor grasps, which the authors suspect come from VLM-estimated friction and the simulator treating objects as rigid.
Limitations
The VLM can only express objectives in the fixed library, and real execution depends on accurate reconstruction and contact dynamics, especially under occlusion and grasp slip. Plans run forward only: once a stage completes, the controller cannot return to it. The evaluation covers four tabletop tasks.
Videos
Figures



Abstract
Vision-language models (VLMs) have shown strong semantic and spatial reasoning capabilities in 3D environments. Recently, connecting VLMs with model-predictive controllers (MPC) has emerged as a paradigm for combining strong semantic reasoning with physics-based planning. However, connecting these two for contact-rich robotic manipulation requires a task representation that bridges the two and captures how physical interactions can be reasoned by VLMs. In this work, we introduce Objectives as Policies (OaP), a framework that connects a pretrained VLM to a shared sampling-based MPC through an editable objective program. Each program specifies a sequence of stages for completing a task, combining running and terminal costs from a cost library with stage completion conditions. Given a natural-language task instruction and scene observation, we build a simulation-based digital twin through real-to-sim, and the VLM generates an objective program that is executed by the MPC in the simulation. The program is iteratively updated by the VLM based on MPC rollouts, before being deployed in the real world. We evaluate our framework in both simulation and the real world on four tabletop manipulation tasks, and show that it outperforms strong baselines. Additional videos and examples are available on the project website: https://zihaolu001.github.io/oap-project/
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
