OpenFlyV2: An onboard vision-language-action model for aerial perception and navigation

📌 Highlights

OpenFlyV2 unifies aerial visual understanding and language-conditioned flight control in one onboard model. The paper's strongest evidence is the combination of competitive aerial perception, simulated navigation gains, measured latency reductions, and zero-shot physical deployment.

• A 0.8B vision-language backbone and 0.2B action expert form a 1-billion-parameter onboard system.
• OpenFlyV2-130K contains 72,352 navigation trajectories and 60,006 VQA pairs across 13 environments.
• The model generates 12-waypoint chunks at 10 Hz on Jetson Orin NX.
• It achieves 62.88% SR in seen simulation, 52.67% SR in unseen simulation, and 48% success across 100 physical trials.
• The main caveat is that real-world success remains below half of trials and was assessed manually.

🎯 Introduction

The paper frames aerial intelligence as a coordination problem rather than a collection of isolated capabilities. A useful aerial robot must interpret scene semantics, ground language-referenced entities, reason about spatial relationships, and translate changing goals into continuous flight behavior. Existing approaches often specialize in one capability, depend on offboard computation, or do not sustain closed-loop operation within onboard latency and memory limits.

OpenFlyV2 targets this gap with a shared vision-language representation connected to a compact action expert. The objective is to preserve broad multimodal competence while adding the temporal context and computational efficiency needed for long-range observations, rapid viewpoint changes, partial observability, and three-dimensional motion.

🔬 Methodology

The architecture combines a 0.8B Qwen3.5-VL backbone with a 0.2B action expert. The backbone receives the current aerial image, instruction, robot state, and MAM tokens. It can either decode a language answer or condition waypoint generation. The action representation is a 12-step chunk with three body-frame motion increments and a stop signal.

MAM forms a bounded history from causal motion events. Action segments are paired with start and end images, encoded through temporal convolution, masked pooling, visual cross-attention, and age embeddings. Equation 1 defines the event token construction. In the reported deployment configuration, MAM reduced the historical-input token count from 446 to at most 218.

MSP clusters normalized nonstationary trajectories with K=4, predicts a motion class and context-dependent adjustments, and samples an initial source chunk using Equation 2. Conditional flow matching then trains and refines the action expert through Equations 3–5. The training process consists of joint VLN/VQA supervision, continued supervised training with MAM and MSP, and reinforcement post-training that updates only the action expert.

📊 Experiments

The evaluation covers aerial vision-language understanding, simulated VLN, ablations, onboard efficiency, and physical flight. Vision-language evaluation used 300 examples each for color recognition, object description, distance estimation, and spatial reasoning, plus 1,000 visual-grounding examples. Simulated navigation used 800 seen and 300 unseen episodes, with success requiring an explicit stop within 25 m of the goal. The baselines were AeroVLA, DualVLN, OpenUAV, and OmniNav.

For aerial understanding, OpenFlyV2 achieved 67.1% visual-grounding Acc@0.5 versus 61.6% for GPT-5.5, distance mean absolute error of 47.32 m versus 52.62 m, relative error of 0.27 versus 0.34, and description quality of 6.38 versus 6.36 out of 10. In simulated navigation, it reached 62.88% SR in seen environments and 52.67% in unseen environments, compared with DualVLN's 38.88% and 36.67%.

The ablations show that MAM improves all four pooled navigation metrics at both one and two flow steps, while MSP is particularly useful at one step. MAM also reduced the historical-input budget by at least 51.1% and reduced action-expert denoising time from 80 ms to 20 ms on Jetson Orin NX. Physical evaluation covered NWPU 02, SJTU 01, NWPU 01, and Shanghai Expo Culture Park; across 50 single-landmark and 50 two-landmark trials, success was 48%.

fig:dataset

Caption: Figure 2. Shared environments and data composition of OpenFlyV2-130K. (A) Thirteen environments supporting both navigation and VQA: four AirSim environments, two Unreal Engine city environments, and seven 3DGS reconstructions of real campuses and post-disaster ruins. (B) Distribution of 72,352 navigation trajectories across the environments. (C) Distribution of 60,006 VQA pairs across color recognition, object description, distance estimation, spatial relationship reasoning, and visual grounding.

Why It Matters: It shows how navigation and visual-language supervision share the same 13-environment data infrastructure while separating trajectory and question-answer coverage.

fig:vl_understanding

Caption: Figure 3. Aerial vision-language understanding and spatial reasoning with OpenFlyV2. (A) Distance estimation for the building marked in green: 179 m compared with a reference distance of approximately 180 m. (B) Description of the building within the yellow box. (C) Relative-distance reasoning identifies the red-box building as closer to the camera than the blue-box building. (D) Localization of a language-described target. The predicted bounding box (dashed orange) and reference box (solid cyan) have an intersection over union (IoU) of 0.753.

Why It Matters: It demonstrates that the same model supports quantitative distance estimation, scene description, relational reasoning, and language-grounded localization.

fig:quantitative

Caption: Figure 4. Quantitative evaluation of aerial understanding and navigation. (A) Comparison with general-purpose vision-language models on color recognition, description generation, visual grounding, distance estimation, and spatial relationship reasoning. Visual grounding is reported as accuracy at an IoU threshold of 0.5 (Acc@0.5), and distance estimation is shown as relative error. (B) Navigation performance in seen and unseen simulated environments, measured by success rate (SR), oracle success rate (OSR), success weighted by path length (SPL), and navigation error (NE). The navigation baselines were trained on our aerial navigation dataset and evaluated on the same 800 seen and 300 unseen...

Why It Matters: It summarizes the central comparison across aerial understanding and simulated navigation, including both seen and unseen environments.

fig:vln

Caption: Figure 5. Onboard aerial vision-language navigation. Three physical flight trials performed by OpenFlyV2 on a DJI M30T, with all navigation inference executed on an onboard NVIDIA Jetson Orin NX.

Why It Matters: It provides qualitative evidence that the system was executed as a fully onboard physical-flight loop rather than only evaluated in simulation.

Figure 6

Caption: Figure 6. Architecture of OpenFlyV2. (A) A shared vision-language backbone encodes the current image, instruction, and memory tokens to condition action generation. (B) Motion-Adaptive Memory (MAM) compresses historical action segments and their visual anchors into compact event-level tokens. (C) The Motion-Structured Prior (MSP) selects and adjusts a data-derived motion prototype to construct the action source distribution. The action expert transforms a sample from this distribution into the final action chunk through conditional flow matching.

Why It Matters: It clarifies the complete data path from current observation and history to motion-conditioned source sampling and final waypoint generation.

🔮 Conclusion

OpenFlyV2 demonstrates that aerial perception, spatial reasoning, language interaction, and continuous flight control can coexist in one onboard model. The combination of shared multimodal features, compressed motion-aware history, structured action initialization, and asynchronous chunk execution enables real-time inference on Jetson Orin NX.

The evidence is strongest for simulated navigation and system feasibility: the model outperforms the tested navigation baselines in simulation, matches or exceeds GPT-5.5 on selected aerial understanding metrics, and transfers without real-world aerial training. Physical success at 48% nevertheless shows that the deployment problem remains open.

🛠️ Future Research Improvements

The paper identifies deviation detection, error recovery, and resuming navigation under changing observations as important next steps. These should be evaluated with explicit recovery-rate, intervention-rate, collision, and time-to-completion measurements rather than aggregate success alone.

Further work should broaden environments and operating conditions, investigate dialogue and reference tracking across requests, and test concurrent navigation and question answering. Larger physical test sets, stronger out-of-distribution evaluation, and safety-assured control would help determine whether the zero-shot simulation-to-flight result generalizes.

🏭 Potential Industry Use Scenarios

A practical near-term use case is language-guided aerial inspection: an operator could specify a landmark or route, then ask questions about the resulting view. The shared representation is suitable for missions that alternate between navigation and visual reporting without switching between separate models.

Other plausible scenarios include infrastructure inspection, post-disaster reconnaissance, campus or facility surveying, and landmark-based search. Deployment would require stronger guarantees around obstacle avoidance, mission completion, operator confirmation, recovery from ambiguous instructions, and operation in environments unlike the rendered training scenes.

💬 Critical Analysis

The paper's main strength is that it measures the full stack: multimodal task competence, simulated navigation, input compression, action-generation latency, onboard execution, and physical flight. The MAM ablation directly supports the claim that motion-conditioned compression can reduce computational cost while improving navigation, and the MSP ablation supports a narrower claim that structured initialization is most useful when only one flow step is available.

The results also reveal important tradeoffs. MAM+MSP reaches 57.09% SR at one step but 57.73% at two steps, while MAM alone reaches 59.45% at two steps. The one-step combined model has OSR of 68.36% versus SR of 57.09%, indicating that entering the goal region does not reliably translate into a valid stop. This gap is especially relevant for builders because stopping and termination logic remain part of the task, not merely an implementation detail.

Reproducibility will depend on access to the promised code, exact simulation assets, annotation filters, baseline training details, and Jetson deployment configuration. The physical study is encouraging but limited by 100 trials, manual success assessment, four locations, and the absence of real-world training data. The next decisive test is whether the same architecture maintains reliable behavior under broader distribution shifts and explicit safety constraints.