A Hierarchical Onboard Vision-Language-Navigation Framework for Aerial Robots in Complex Environments
TL;DR VLN-AR runs a fine-tuned vision-language model on a drone's onboard computer to turn language instructions into 3D waypoints, which a classical planner then flies collision-free. It completes 91 of 101 real-world tasks and reaches 78.15% success on the AerialVLN benchmark.
What they do
VLN-AR lets a small drone follow open-vocabulary instructions such as “head to the black chair, then the front circular pillar, and finally land on the carpet”, with all inference on an onboard NVIDIA Jetson Orin NX. A vision-language model decides where to go and a classical planner decides how to fly there. Most training data come from an automated pipeline that flies a simulated drone through 3D Gaussian Splatting (3DGS) reconstructions of real scenes.
How it works
- Hierarchy. A post-trained Qwen3-VL model sees the current frame and 8 history frames (4 s) and outputs a 3D waypoint with yaw, a 2D bounding box or chain-of-thought, plus replan and finish flags. An action expert built on Ego-Planner turns the target into a collision-free trajectory at over 200 Hz; the VLM runs at 2 Hz.
- Data. Videos of real indoor and outdoor scenes become 3DGS scenes in Unity. Tasks are sampled from free space and flown by an optimization-based planner, and an MLLM rewrites the instructions. In total about 4 million samples, including VQA, grounding and spatial-reasoning data.
- Training. Grounding-and-reasoning SFT, then navigation SFT, then GRPO reinforcement fine-tuning in closed-loop simulation with a sparse success reward.
- Agent and speed-up. An agent splits long instructions into subtasks, keeps a memory of frames and trajectories, and rotates 360° or reasons about where to explore when the target is not visible. Instruction-guided pruning keeps 25% of vision tokens, and quantization plus kernel optimizations make decoding 7.1x-9.4x faster.
Results
- Simulation: 78.15% success on AerialVLN and 76.88% on OpenFly (neither used for training), against 67.37% and 64.67% for the authors’ earlier VLA-AN and under 50% for every other baseline, including π0, GR00T N1.5 and Gemini 3 Pro.
- Real world: 91 of 101 tasks succeed on two drones in homes, offices, logistics facilities, parks and shopping districts, with no failures caused by collisions. Robustness tests (dim light, people walking into the path, unseen scenes) succeed 17-19 times out of 20.
- Without the agent, long-horizon success drops from 88.0% to 57.4%. Training on 3DGS data works about as well as training on real-world data (long-horizon 88.0% vs 88.7%, both mixed with diverse data).
- Onboard inference time per step drops from about 4100 ms to 494 ms.
Limitations
Most real-world failures come from inaccurate goal positions: projecting small or distant targets from 2D to 3D, ordinal instructions such as “the fifth tree from near to far”, and ambiguous instructions in scenes with look-alike objects. Motion blur from the RealSense D435 also causes failures.
Videos
Figures




Abstract
Unmanned Aerial Vehicles (UAVs) are expected to achieve semantic reasoning, intelligent decision-making, and long-horizon navigation under strict onboard constraints. While recent foundation-model-driven approaches have shown promising progress toward this goal, existing methods remain limited by challenges in 3D data availability, trajectory feasibility, long-horizon reasoning, and computational efficiency in complex environments. To bridge these gaps, we propose VLN-AR, a hierarchical vision-language-navigation (VLN) framework for aerial robots. VLN-AR decouples multimodal reasoning and low-level control, enabling structured prediction of 3D pose commands and navigation intentions that are translated into dynamically feasible actions. To mitigate data scarcity and domain gaps, we introduce a fully automated 3D Gaussian Splatting (3DGS)-based pipeline to generate scalable photorealistic UAV navigation data. A progressive three-stage training strategy further enhances grounding, scene reasoning, and complex navigation. In addition, a lightweight cognitive agent improves long-horizon reasoning via memory-aware task decomposition. Finally, we design onboard optimization techniques, including token sparsification and model lightweighting, achieving 7.1x-9.4x speedup while maintaining high navigation performance on resource-limited UAV platforms. Experiments show great improvements over existing methods in both simulation and real-world UAV navigation. Project website: https://wyz000.github.io/VLN-AR/
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
