← Paper Digest

Objectives as Policies: Vision-Language Model Cost Programs for Contact-Rich Manipulation

Zihao Lu, Trong Gia Hung Nguyen, Fuquan Wang, Yifan Zhu

November 2026

Real robot: picking up the eraser and placing it on the box.

TL;DR A VLM writes a multi-stage cost program for a sampling-based MPC from a fixed library of cost terms, refines it on rollouts in a real-to-sim digital twin, and the same program then runs on the real arm. Across four contact-rich tabletop tasks with moved objects, programs that succeed in simulation succeed in 32 of 40 real executions.

What they do

Contact-implicit MPC handles contact well but needs costs engineered for each task and scene; vision-language-action models bring semantics but no physics model. Objectives as Policies (OaP) has a VLM write the MPC’s objective as an editable program, tests it in a digital twin of the scene, revises it, and then runs it on a Flexiv Rizon 4s arm with a parallel gripper. No task-specific training.

How it works

  • Objective program. A sequence of stages, each with running costs, terminal costs, weights and completion thresholds, built from a fixed library of residuals: distances, axis alignment, rotation progress, region membership, tool-mediated contact, object held, gripper command. A stage verifier advances once every terminal residual is under its threshold.
  • Real-to-sim. Meshes from SAM 3D, scale and pose refined with Any6D, density and friction estimated by a VLM; FoundationPose tracks objects on hardware.
  • MPC. MPPI with 1000 rollouts over 600 physics steps in MuJoCo-warp, replanning every 40 controls.
  • Refinement. GPT-5.6-sol drafts and self-reviews the program, then revises it from six subsampled state records of each failed simulated rollout.

Results

  • Simulation (four tasks, 10 trials each): self-review raises first-try success from 20/40 to 26/40, and up to five feedback rounds reach 39/40.
  • With two feedback rounds OaP succeeds in 36/40, against 31/40 for an adapted CoRAL, 17/40 for ReKep, 11/40 for Language to Rewards, 10/40 for pi0.5 and 0/40 for OpenVLA-OFT (released checkpoints, no fine-tuning).
  • Without the cost library, first-execution success with self-review falls from 65% to 35%.
  • Real robot: 32/40 executions with object positions perturbed. Most failures were poor grasps, which the authors suspect come from VLM-estimated friction and the simulator treating objects as rigid.

Limitations

The VLM can only express objectives in the fixed library, and real execution depends on accurate reconstruction and contact dynamics, especially under occlusion and grasp slip. Plans run forward only: once a stage completes, the controller cannot return to it. The evaluation covers four tabletop tasks.

Videos

Real robot: grasping the cup and tilting it.
Real robot: grasping the eraser and using it to push the box.

Figures

From Figure 1: The semantic understanding of a VLM and physics-based MPC, interfaced through a structured objective p...
From Figure 1: The semantic understanding of a VLM and physics-based MPC, interfaced through a structured objective program for multi-stage tasks. Bottom: real-world executions of the four tasks, pick-and-place, cup reorientation, direct push and push with tool.
From Figure 2: Method overview. A digital twin is built from the scene through real-to-sim. The VLM writes an objecti...
From Figure 2: Method overview. A digital twin is built from the scene through real-to-sim. The VLM writes an objective program (stages with running costs, terminal costs and completion thresholds) from the task, the twin and a cost library, and improves it from MPC rollouts in the twin. The program is then deployed in the real world with the same MPC and stage verifier while the twin is synced to the real scene.
From Figure 3: Real-world execution of the four tasks, early, middle and late in each run. Each pair shows a real cam...
From Figure 3: Real-world execution of the four tasks, early, middle and late in each run. Each pair shows a real camera frame (left) and its MuJoCo reconstruction (right) from the recorded robot and object states.
Abstract

Vision-language models (VLMs) have shown strong semantic and spatial reasoning capabilities in 3D environments. Recently, connecting VLMs with model-predictive controllers (MPC) has emerged as a paradigm for combining strong semantic reasoning with physics-based planning. However, connecting these two for contact-rich robotic manipulation requires a task representation that bridges the two and captures how physical interactions can be reasoned by VLMs. In this work, we introduce Objectives as Policies (OaP), a framework that connects a pretrained VLM to a shared sampling-based MPC through an editable objective program. Each program specifies a sequence of stages for completing a task, combining running and terminal costs from a cost library with stage completion conditions. Given a natural-language task instruction and scene observation, we build a simulation-based digital twin through real-to-sim, and the VLM generates an objective program that is executed by the MPC in the simulation. The program is iteratively updated by the VLM based on MPC rollouts, before being deployed in the real world. We evaluate our framework in both simulation and the real world on four tabletop manipulation tasks, and show that it outperforms strong baselines. Additional videos and examples are available on the project website: https://zihaolu001.github.io/oap-project/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.