Grounding Object-to-Object Hazard Reasoning into 3D Risk Fields for Task-Conditioned Safety
TL;DR From one RGB-D image and the task, a VLM decides whether to refuse the task or which object pairs to keep apart, and the hazards become 3D risk fields that steer a Franka's motion. In simulation it reaches 81.4% safe success against at most 51.7% for baselines, without ever refusing a safe task.
What they do
Some manipulation hazards come from how objects interact: carrying coffee over a laptop risks a spill, and a metal cup in a microwave is a fire hazard whatever the path. A VLM sees the scene and the task and either refuses the task or names object pairs to keep apart. Those hazards become 3D risk fields that an obstacle-avoidance filter uses to correct the robot’s nominal motion.
How it works
- Hazard reasoning as tool calls. The VLM (GPT-4o) calls one function to reject a task, or functions that name a held object, an anchor object and a relation to avoid: above, below, inside, between or collision.
- Grounding. SAM-3 masks and depth give the anchor’s point cloud, extruded by the relation (a pillar for “avoid above”).
- Risk field. The points are voxelized and smoothed into a density, rescaled so the field equals 1 on the hazard surface, below 1 inside and above 1 outside.
- Execution. Obstacle modulation checks the field at the held object’s bounding-box corners and face centers and bends a waypoint-based nominal trajectory around the hazard.
Results
- Simulation (Franka Panda in CoppeliaSim, 9 scenes, 10 randomizations each, 3 runs): average safe success 81.4% against 45.6% (no context), 34.5% (text-only scene) and 51.7% (plain obstacle avoidance), with the lowest violation rate (4.4% of the path).
- Task rejection: all variants refuse the 96 non-recoverable runs, but the method falsely rejects 0 of 183 preventable-hazard and safe runs, against 54 with a text scene description plus tools and 105 with task text alone.
- With GPT-4o, the toolset cuts false rejection from 79.8% to 0% and hazard misses from 25.0% to 7.5% compared with prompting.
- Real Franka, 5 runs per scene: safe success 5/5 on laptop-cup and 4/5 on steak transfer, against 1/5 for both baselines. The real rejection scene was refused in every run.
Limitations
Hazards are limited to a fixed set of spatial relations; ones that depend on time or earlier actions are out of scope. The risk field comes from a single RGB-D image taken at the start. In simulation the method completes the steak task less often than the baselines (75.8% against 100%), and the real-world test covers three scenes with 5 runs each.
Videos
Figures




Abstract
Safety in robotics does not solely depend on geometric constraints, but also on semantics: the given task, environment properties, and interactions between objects in the scene. We present a framework based on vision-language models (VLMs) to infer task- and scene-dependent safety constraints for robot manipulation. We provide the VLM a structured safety tool based on an RGB-D observation and a natural-language task instruction that first decides whether the task should be rejected since its execution may result in a non-recoverable hazardous outcome. Otherwise, it identifies preventable object-pair hazards and specifies the spatial relations that must be avoided. Open-vocabulary segmentation and depth information are then used to ground the detected objects in 3D and construct continuous risk fields that represent the corresponding hazardous regions. These fields are integrated with an obstacle avoidance module that corrects nominal robot motions during execution while accounting for the risk field. We evaluate the framework on preventable hazards, tasks requiring rejection, and safe variants in simulation and the real-world. Project website: https://vlm-hazard-riskfield.github.io
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
