Robot Navigation and Obstacle Avoidance from Randomly Positioned External Cameras

TL;DR VINE lets a robot navigate a room from randomly placed external cameras by predicting a top-down obstacle map from their images and handing it to a PPO policy or an A* planner. In simulation it reaches 92-95% success, above PPO with an onboard camera (83.9%).
What they do
Indoor rooms often have fixed cameras that see the whole layout at once. The authors ask whether a robot can navigate to a point goal using only such external cameras. Plain PPO, which works with an onboard camera, fails here: the external images say little about the robot’s own position and heading, and within their training budget the reward alone does not teach the mapping to egocentric actions. VINE instead learns, with supervision, to predict a top-down obstacle map from the external images and hands that map to a controller.
How it works
- Obstacle map network. A ResNet-34 encoder and UNet decoder turn each external image into a 64×64 obstacle map of the room. FiLM layers condition it on the approximate camera pose and the room dimensions, and it gets no camera intrinsics. With two cameras, features are fused using learned per-camera weights.
- Training data. 80,000 Isaac Lab snapshots of randomized rectangular rooms, with randomized camera position, look-at target, zoom, pose noise and pixel noise.
- Controllers. Either a PPO policy acting on the map warped into the robot’s frame, or A* on the global map. The two stages share no gradients, so the controller needs no retraining when the map network changes.
Results
- Map IoU is 0.909 with one camera and 0.913 with two. Projection baselines built on Grounding DINO, SAM 2.1 and Depth Anything V2 reach at most 0.80 without camera noise and 0.53 or less with it.
- Without FiLM conditioning IoU drops to 0.592, and without on-the-fly domain randomization it drops to 0.570.
- Navigation success is 92.3% with PPO and one camera and 94.4-95.1% with A*. PPO with an onboard camera gets 83.9%, and PPO trained directly on a fixed external camera gets 24.7%.
- Moved to photorealistic rooms with no policy retraining, VINE succeeds 90.8% of the time with known furniture and 77.2% with furniture unseen in training.
Limitations
Simulation only, with real-robot experiments left to future work. VINE needs the robot pose, an approximate camera pose and the room dimensions at run time, assumes rectangular rooms, and misses obstacles hidden behind other obstacles. Map accuracy falls sharply once the camera moves past its training range (IoU 0.48 at 3 m, against a 2 m training range).
Figures



Abstract
We study robot navigation in indoor environments from fixed exocentric cameras (infrastructure-mounted, external to the robot). We show that standard Proximal Policy Optimization (PPO), which succeeds with onboard cameras, fails to learn this navigation end-to-end in our setup. This failure results from the perspective mismatch between exocentric observations and egocentric action and goal formulations, which is too complex to resolve from the reward signal alone. To address this, we propose Viewpoint-Invariant Navigation from Exocentric Cameras (VINE). VINE uses an obstacle map estimation network, trained with supervised learning, that takes exocentric images, camera pose, robot pose, and room dimensions as input and produces a viewpoint-invariant obstacle map. This map can be used either by a PPO policy or by a planner such as A*. VINE's map estimation performs better than perspective mapping approaches, handles cameras placed across a wide range of positions and orientations without view-specific tuning, requires only approximate camera pose estimates (noise with a standard deviation of 30cm per axis in x, y, and z and about 20° of rotation error on average), is robust to intrinsic variation, and extends to two-camera setups with partial coverage. Despite operating from cameras randomized in position during evaluation, VINE surpasses the navigation success rate of PPO with an onboard camera in simulated environments.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
