← Paper Digest

Grounding Object-to-Object Hazard Reasoning into 3D Risk Fields for Task-Conditioned Safety

Mariana Sosa Guzmán, Yide Song, Amin Abyaneh, Gregory Dudek, Hsiu-Chin Lin

November 2026

Real robot: plain obstacle avoidance carries the coffee cup over the laptop; the VLM method routes around it.

TL;DR From one RGB-D image and the task, a VLM decides whether to refuse the task or which object pairs to keep apart, and the hazards become 3D risk fields that steer a Franka's motion. In simulation it reaches 81.4% safe success against at most 51.7% for baselines, without ever refusing a safe task.

What they do

Some manipulation hazards come from how objects interact: carrying coffee over a laptop risks a spill, and a metal cup in a microwave is a fire hazard whatever the path. A VLM sees the scene and the task and either refuses the task or names object pairs to keep apart. Those hazards become 3D risk fields that an obstacle-avoidance filter uses to correct the robot’s nominal motion.

How it works

  • Hazard reasoning as tool calls. The VLM (GPT-4o) calls one function to reject a task, or functions that name a held object, an anchor object and a relation to avoid: above, below, inside, between or collision.
  • Grounding. SAM-3 masks and depth give the anchor’s point cloud, extruded by the relation (a pillar for “avoid above”).
  • Risk field. The points are voxelized and smoothed into a density, rescaled so the field equals 1 on the hazard surface, below 1 inside and above 1 outside.
  • Execution. Obstacle modulation checks the field at the held object’s bounding-box corners and face centers and bends a waypoint-based nominal trajectory around the hazard.

Results

  • Simulation (Franka Panda in CoppeliaSim, 9 scenes, 10 randomizations each, 3 runs): average safe success 81.4% against 45.6% (no context), 34.5% (text-only scene) and 51.7% (plain obstacle avoidance), with the lowest violation rate (4.4% of the path).
  • Task rejection: all variants refuse the 96 non-recoverable runs, but the method falsely rejects 0 of 183 preventable-hazard and safe runs, against 54 with a text scene description plus tools and 105 with task text alone.
  • With GPT-4o, the toolset cuts false rejection from 79.8% to 0% and hazard misses from 25.0% to 7.5% compared with prompting.
  • Real Franka, 5 runs per scene: safe success 5/5 on laptop-cup and 4/5 on steak transfer, against 1/5 for both baselines. The real rejection scene was refused in every run.

Limitations

Hazards are limited to a fixed set of spatial relations; ones that depend on time or earlier actions are out of scope. The risk field comes from a single RGB-D image taken at the start. In simulation the method completes the steak task less often than the baselines (75.8% against 100%), and the real-world test covers three scenes with 5 runs each.

Videos

Real robot: the raw steak is moved to the cutting board without passing over the bread.

Figures

From Figure 1: Overview. Given a task instruction and a scene image, the VLM either rejects the task or identifies ob...
From Figure 1: Overview. Given a task instruction and a scene image, the VLM either rejects the task or identifies object-pair hazards, which are grounded into a 3D risk field that guides safe execution.
From Figure 3: Risk field for the laptop under an avoid-above hazard. The field value is below 1 inside the hazard re...
From Figure 3: Risk field for the laptop under an avoid-above hazard. The field value is below 1 inside the hazard region, 1 on its surface (red line) and above 1 outside.
From Figure 6: Simulated laptop-cup runs, top view. No context fails the task, Text only and Simple OA carry the cup ...
From Figure 6: Simulated laptop-cup runs, top view. No context fails the task, Text only and Simple OA carry the cup over the laptop (22.6% and 21.6% violation), and the VLM method completes it with no violation.
From Figure 8: Real-world steak transfer, VLM method against Simple OA, from the top camera and in top, side and fron...
From Figure 8: Real-world steak transfer, VLM method against Simple OA, from the top camera and in top, side and front views of the scene with the avoid-above risk regions over the bread and cheese.
Abstract

Safety in robotics does not solely depend on geometric constraints, but also on semantics: the given task, environment properties, and interactions between objects in the scene. We present a framework based on vision-language models (VLMs) to infer task- and scene-dependent safety constraints for robot manipulation. We provide the VLM a structured safety tool based on an RGB-D observation and a natural-language task instruction that first decides whether the task should be rejected since its execution may result in a non-recoverable hazardous outcome. Otherwise, it identifies preventable object-pair hazards and specifies the spatial relations that must be avoided. Open-vocabulary segmentation and depth information are then used to ground the detected objects in 3D and construct continuous risk fields that represent the corresponding hazardous regions. These fields are integrated with an obstacle avoidance module that corrects nominal robot motions during execution while accounting for the risk field. We evaluate the framework on preventable hazards, tasks requiring rejection, and safe variants in simulation and the real-world. Project website: https://vlm-hazard-riskfield.github.io

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.