← Paper Digest

Incorporating World Models in Decision Making for Next Best View Selection in Active Sensing

Nikita Sarawgi, Omey M. Manyar, Satyandra K. Gupta

November 2026

TL;DR A small Gaussian-process adapter learns where a structured-light sensor returns no depth, so next-best-view selection with a frozen pretrained world model stops picking views that come back empty. On a held-out reflective helmet it reaches 98.4% surface coverage in 4 captures; nominal selection is still at 97.5% after 10.

What they do

A pretrained world model can predict which surface a robot-mounted depth sensor would see from a candidate pose. A visible surface does not always give valid depth, though: on reflective parts a structured-light sensor drops out. The authors keep the predictor frozen and add a small deployment adapter, learned from local calibration captures, that estimates where the sensor will return no depth. Next-best-view selection then counts only predicted surface that the sensor is likely to measure.

How it works

  • Fixed predictor. From 6 posed context images, FrameCrafter predicts the RGB view at each candidate pose and Depth Anything 3 turns it into a predicted point cloud.
  • Adapter. A sparse variational Gaussian-process classifier maps viewing distance, incidence angle and normalized intensity to a dropout probability. Its labels come from comparing calibration captures with a dense reference surface.
  • Selection. Predicted points with dropout probability above 20% are discarded. A greedy rule then picks the pose that covers the most unmeasured voxels.

Results

  • Setup: a FANUC M-710iC with a Zivid 2+ structured-light sensor. The adapter is trained on 50 captures of four helmets and tested on a held-out reflective blue helmet.
  • Deployment-aware selection reaches 98.4% coverage in 4 captures. Nominal selection reaches 95.8% after 4 and 97.5% after 10, short of the 98% target. That is 60% fewer captures.
  • After the second capture, nominal selection realizes 11.7% of its predicted coverage gain, against 114.8% with the adapter. Nominal selection keeps choosing views of a lower-edge region that never returns depth.

Limitations

This is a short paper. The test covers one held-out helmet, with one run per method under fixed lighting and sensor settings. For this experiment the authors used depth projected from the reference scan instead of the predicted depth, which they say still needs alignment. Calibration requires reference surfaces. They have not yet tested when changes in material, sensor or lighting require retraining the adapter.

Figures

From Figure 1: Deployment-aware next-best-view selection. A fixed predictor supplies geometry; a locally learned adap...
From Figure 1: Deployment-aware next-best-view selection. A fixed predictor supplies geometry; a locally learned adapter estimates sensor return for action scoring. Captures update the reconstruction.
From Figure 2: (a) FrameCrafter RGB predictions at 4 candidate views of the held-out blue helmet. (b) Measured recons...
From Figure 2: (a) FrameCrafter RGB predictions at 4 candidate views of the held-out blue helmet. (b) Measured reconstructions: nominal selection leaves a lower-edge gap after 10 captures (97.5% coverage), while deployment-aware selection recovers it within 4 (98.4%).
Abstract

Efficient robotic 3D reconstruction requires selecting views that contribute new surface measurements with few physical captures. Pretrained world models can guide this choice, but a predicted surface may not produce valid depth measurements under the operating conditions of the deployed sensor. We present a deployment adaptation methodology for next-best-view selection that learns a sensor-return adapter from limited local calibration data while keeping the pretrained predictor fixed. The adapter estimates measurement dropout over each predicted surface, allowing the selection method to favor views likely to fill gaps in the measured reconstruction. In a real-robot experiment with a structured-light sensor, the adapted method achieves higher measured surface coverage with 60% fewer captures than nominal selection. The adapted method recovers a surface region that the nominal method repeatedly targets but fails to measure. These results suggest that robots can reuse pretrained predictive models while adapting their decisions to local operating conditions in data-scarce environments.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.