← Paper Digest

A Hierarchical Onboard Vision-Language-Navigation Framework for Aerial Robots in Complex Environments

Yuze Wu, Mo Zhu, Xingxing Li, Yuheng Du, Zhaoqi Wang, Yuxin Fan, Wenjun Li, Kenghou Hoi, Zhichao Han, Xin Zhou, Fei Gao

November 2026

Indoor flight at 4x speed through a multi-step instruction to find several people and check the TV.

TL;DR VLN-AR runs a fine-tuned vision-language model on a drone's onboard computer to turn language instructions into 3D waypoints, which a classical planner then flies collision-free. It completes 91 of 101 real-world tasks and reaches 78.15% success on the AerialVLN benchmark.

What they do

VLN-AR lets a small drone follow open-vocabulary instructions such as “head to the black chair, then the front circular pillar, and finally land on the carpet”, with all inference on an onboard NVIDIA Jetson Orin NX. A vision-language model decides where to go and a classical planner decides how to fly there. Most training data come from an automated pipeline that flies a simulated drone through 3D Gaussian Splatting (3DGS) reconstructions of real scenes.

How it works

  • Hierarchy. A post-trained Qwen3-VL model sees the current frame and 8 history frames (4 s) and outputs a 3D waypoint with yaw, a 2D bounding box or chain-of-thought, plus replan and finish flags. An action expert built on Ego-Planner turns the target into a collision-free trajectory at over 200 Hz; the VLM runs at 2 Hz.
  • Data. Videos of real indoor and outdoor scenes become 3DGS scenes in Unity. Tasks are sampled from free space and flown by an optimization-based planner, and an MLLM rewrites the instructions. In total about 4 million samples, including VQA, grounding and spatial-reasoning data.
  • Training. Grounding-and-reasoning SFT, then navigation SFT, then GRPO reinforcement fine-tuning in closed-loop simulation with a sparse success reward.
  • Agent and speed-up. An agent splits long instructions into subtasks, keeps a memory of frames and trajectories, and rotates 360° or reasons about where to explore when the target is not visible. Instruction-guided pruning keeps 25% of vision tokens, and quantization plus kernel optimizations make decoding 7.1x-9.4x faster.

Results

  • Simulation: 78.15% success on AerialVLN and 76.88% on OpenFly (neither used for training), against 67.37% and 64.67% for the authors’ earlier VLA-AN and under 50% for every other baseline, including π0, GR00T N1.5 and Gemini 3 Pro.
  • Real world: 91 of 101 tasks succeed on two drones in homes, offices, logistics facilities, parks and shopping districts, with no failures caused by collisions. Robustness tests (dim light, people walking into the path, unseen scenes) succeed 17-19 times out of 20.
  • Without the agent, long-horizon success drops from 88.0% to 57.4%. Training on 3DGS data works about as well as training on real-world data (long-horizon 88.0% vs 88.7%, both mixed with diverse data).
  • Onboard inference time per step drops from about 4100 ms to 494 ms.

Limitations

Most real-world failures come from inaccurate goal positions: projecting small or distant targets from 2D to 3D, ordinal instructions such as “the fifth tree from near to far”, and ambiguous instructions in scenes with look-alike objects. Motion blur from the RealSense D435 also causes failures.

Videos

The drone follows a person in a brown coat at a safe distance (3x speed).

Figures

From Figure 1: Overview of VLN-AR. (I) An automated pipeline collects aerial navigation data in 3DGS scenes. (II) A t...
From Figure 1: Overview of VLN-AR. (I) An automated pipeline collects aerial navigation data in 3DGS scenes. (II) A three-stage training recipe strengthens spatial grounding, scene reasoning and long-horizon navigation. (III) Onboard, VLN-AR turns language instructions and camera images into collision-free navigation commands.
From Figure 3: The VLN-AR architecture. A post-trained VLM reads history and current frames plus the instruction and ...
From Figure 3: The VLN-AR architecture. A post-trained VLM reads history and current frames plus the instruction and outputs task flags, a 3D pose, a 2D box or chain-of-thought; an action expert turns these into safe 3D action sequences.
From Figure 9: The two test drones (NCS-alpha with a RealSense D455, NCS with a D435) and their perception, computati...
From Figure 9: The two test drones (NCS-alpha with a RealSense D455, NCS with a D435) and their perception, computation and motion software stack.
From Figure 12: Success rate on object navigation, long-horizon navigation and spatial grounding when training on mes...
From Figure 12: Success rate on object navigation, long-horizon navigation and spatial grounding when training on mesh, real-world or 3DGS data, alone or mixed with diverse data (grounding, reasoning, spatial VQA).
Abstract

Unmanned Aerial Vehicles (UAVs) are expected to achieve semantic reasoning, intelligent decision-making, and long-horizon navigation under strict onboard constraints. While recent foundation-model-driven approaches have shown promising progress toward this goal, existing methods remain limited by challenges in 3D data availability, trajectory feasibility, long-horizon reasoning, and computational efficiency in complex environments. To bridge these gaps, we propose VLN-AR, a hierarchical vision-language-navigation (VLN) framework for aerial robots. VLN-AR decouples multimodal reasoning and low-level control, enabling structured prediction of 3D pose commands and navigation intentions that are translated into dynamically feasible actions. To mitigate data scarcity and domain gaps, we introduce a fully automated 3D Gaussian Splatting (3DGS)-based pipeline to generate scalable photorealistic UAV navigation data. A progressive three-stage training strategy further enhances grounding, scene reasoning, and complex navigation. In addition, a lightweight cognitive agent improves long-horizon reasoning via memory-aware task decomposition. Finally, we design onboard optimization techniques, including token sparsification and model lightweighting, achieving 7.1x-9.4x speedup while maintaining high navigation performance on resource-limited UAV platforms. Experiments show great improvements over existing methods in both simulation and real-world UAV navigation. Project website: https://wyz000.github.io/VLN-AR/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.