VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding

📌 Highlights

VFlash targets a specific cost in physical AI systems: visual plans become slower to produce as they include more coordinates, waypoints, objects, or dimensions. Its central idea is to decode a complete visual primitive in parallel once its structure is known. A tool call can specify a variable length, while the primitive type defines coordinate roles and boundaries.

The paper’s main empirical pattern is that spatial quality need not be measured by reproducing the AR target’s exact token sequence. The drafter is trained on annotated primitives and accepted when its output meets structural checks. On the evaluated robot trajectory tasks, VFlash reduces errors versus the shared AR target. In the MolmoSpaces setup, the VFlash-conditioned WAM reports 70.2 average success versus 62.6 for its baseline. Separately, the paper reports approximately 25× faster single-rollout generation on PointMotionBench; this speed claim is specific to that reported setting.

• Parallelize coordinates at the complete-primitive level, with schema-dependent block length.
• Retain verified speculative decoding for text while directly accepting valid visual primitives.
• Treat speed and spatial utility as distinct from token agreement; box results show a quality-speed trade-off between direct acceptance and AR fallback.

🎯 Introduction

The paper frames VLMs as high-level components that interpret images and instructions for robotics and autonomous driving. Their outputs can include visual plans: points and boxes that identify locations or objects, and trajectories that describe intended movement or future scene motion. These structured outputs make spatial goals explicit before action. The cited prior approach generates coordinates sequentially, so increasing trajectory length, object count, or joint 2D and 3D detail increases decoding work and can limit update frequency under a fixed latency budget.

Parallel decoding and speculative methods reduce sequential work, but ordinary speculative verification accepts only a draft prefix that agrees with the target. The paper argues that token agreement is not an adequate criterion for spatial usefulness: a small coordinate difference can fail exact token verification even when geometric error is small. Existing parallel visual decoding cited in the paper is described as limited to fixed-size 2D primitives. VFlash’s objective is to align training and decoding to complete primitive boundaries, support diverse structures and variable lengths, and directly accept structurally valid spatial outputs while retaining verification for text.

🔬 Methodology

VFlash’s FPD uses a frozen target VLM and a shallow non-causal drafter. The schema describes primitive type, coordinate dimensions, and extent. Fixed-size boxes can be determined by type; a variable-length trajectory additionally uses a tool call to declare extent. This means the decoder knows the primitive’s boundaries and coordinate layout before predicting its values. The model can represent joint 2D image-plane and 3D metric coordinates in one primitive, and the paper also describes dense motion blocks. Figure 2 illustrates the trigger and schema-dependent block design.

For each block, the observed start token is followed by masked positions. The drafter uses fused features from selected target layers as context and predicts the positions jointly. Equation 3 states the factorized parallel prediction over positions. Training first adapts the AR VLM to the spatial response format, freezes it, then trains the drafter on annotated primitive blocks. Equation 4 is a weighted cross-entropy over those positions; labels come from annotations or rendered coordinate grids, not the target’s generated continuation. The paper says primitive positions use uniform weights. Text-containing examples also train the ordinary drafting path.

At inference, decoding uses 16-position DFlash blocks until a primitive-start token is accepted. The decoder resolves the schema, drafts the complete block in one pass, and applies vocabulary constraints for coordinates and delimiters plus cross-position rules. A trajectory that passes its schema validator is committed directly. Boxes also use a spatial-confidence guard; uncertain coordinates can trigger constrained AR regeneration. Equation 5 specifies the accepted-token count for direct acceptance or partial fallback, while Equation 6 expresses valid-primitive latency amortized over block length. Text remains on target-verified speculative decoding. These choices make the acceptance policy part of the method, not simply a change in drafter architecture.

📊 Experiments

The paper evaluates open-loop 2D trajectories on ShareRobot-Bench and VABench-V using DFD, HD, and RMSE in normalized image coordinates, all lower-is-better. The controlled comparison uses the same 4B target for Baseline (AR) and VFlash. VFlash lowers all six errors: for example, VABench-V RMSE is 0.089 versus 0.093, and ShareRobot-Bench HD is 0.162 versus 0.175. The authors report reductions of 7% on HD and 3–6% on DFD and RMSE across the two benchmarks. Table 2 preserves the paired values.

For category-conditioned boxes, the datasets are COCO and Dense200. The metrics are F1 at IoU 0.50 and 0.95, mean F1, and boxes per second (BPS), with higher values better. Direct VFlash accepts schema-valid boxes without verification; the fallback variant regenerates boxes rejected by the spatial-confidence guard. On COCO, fallback mean F1 is 47.79 versus 48.68 for AR, with 8.76 versus 3.15 BPS. On Dense200, fallback mean F1 is 61.78 versus 60.10, with 17.46 versus 6.73 BPS. Direct decoding reaches 22.66 BPS on Dense200 but mean F1 is 39.09 versus 60.10 for AR. This result illustrates why throughput and quality must be reported together.

For open-loop 3D robot trajectories, Table 4 evaluates InternA1, AgiBot, and MolmoBot. ADE and FDE are in meters and lower is better; SR@5 and SR@10 count waypoint fractions within 5 cm and 10 cm and higher is better. Across the three training variants—3D-only, 2D+3D, and 2D+3D with wrist-camera training—the VFlash decoder improves all four metrics over its AR counterpart for all three datasets. The evidence notes that evaluation uses single-view inputs even for the wrist-camera training variant. For dense 3D point motion, PointMotionBench comprises HOT3D, WorldTrack, and DAVIS; ADE/FDE are lower-is-better and PWT is higher-is-better. VFlash has mixed results against MolmoMotion-AR: it improves DAVIS ADE/FDE and HOT3D FDE, but WorldTrack FDE increases and PWT is lower across the three datasets. The paper reports approximately 25× faster single-rollout generation for this task.

Downstream evaluations cover MolmoSpaces, the LIBERO family (LIBERO, LIBERO-Plus, LIBERO-PRO), and autonomous driving on NAVSIM and the Waymo test split. On MolmoSpaces, the same WAM’s average success is 70.2 with VFlash plans versus 62.6 for its baseline; Table 7 reports 61.0 for AR-generated 2D+3D plans, 69.3 for VFlash 3D-only plans, and 63.7 for the image-context variant. Figure 3 compares the LIBERO-family success rates. NAVSIM uses NC, DAC, EP, TTC, comfort, and PDMS, while Waymo uses ADE at 3 and 5 seconds and RFS; the table marks metric directions. VFlash-RL reports 90.7 PDMS on NAVSIM and 7.86 RFS on Waymo. The paper also reports per-step latency reductions across Jetson Thor, DGX Spark, and H100, with the exact reduction varying by task and platform.

Figure 1

Caption: Figure 1: VFlash is a generalist decoding framework for fast, accurate visual planning across Physical AI tasks such as robotics and autonomous driving. By aligning prediction blocks with the known structure of visual primitives, it enables parallel decoding of common structured outputs such as bounding boxes, variable-length 2D/3D trajectories, and dense point motion, while retaining verified speculative decoding for text and sub-task plans. Compared with autoregressive (AR) decoding, VFlash improves accuracy on 3D robot trajectory planning while substantially reducing decoding latency across NVIDIA platforms, including Jetson Thor, DGX Spark, and H100.

Why It Matters: The overview connects primitive decoding to the paper’s accuracy and latency claims and shows the stated deployment platforms and output types.

Figure 2

Caption: Figure 2: FPD design. A primitive-start token triggers a prediction block layout determined by the primitive type and the preceding tool calling. By adapting the prediction block to each primitive’s structure, VFlash supports different spatial formats and lengths without requiring token-level agreement with the VLM.

Why It Matters: It illustrates how tool calls, primitive type, and block length determine the drafter’s output layout.

🔮 Conclusion

The paper presents VFlash as a way to replace sequential coordinate generation with parallel prediction over known visual primitive structures. Its reported evidence includes lower open-loop robot trajectory errors than the shared AR target, task-dependent gains in downstream plan-conditioned manipulation and driving, and reduced latency on evaluated devices. The authors’ stated future direction is to test whether multiple complementary primitives generated from shared observations can improve control within a fixed latency budget.

The results support the premise that spatial outputs can be evaluated for their geometry and downstream usefulness rather than exact token agreement. The evidence is encouraging but task-specific: box quality depends on fallback policy, and dense-motion metrics show mixed comparisons against AR.

🛠️ Future Research Improvements

The authors explicitly propose examining whether multiple complementary primitives from shared observations improve control within a fixed latency budget. That direction could test whether a planner benefits from generating alternatives or coordinated spatial descriptions, while accounting for the extra decoding and action-selection cost.

Further evaluation should characterize the boundary between a structurally valid primitive and a useful plan. The reported box comparison already shows that direct acceptance can increase throughput while lowering mean F1, particularly on Dense200; the fallback policy changes that balance. Testing acceptance and fallback behavior across harder scenes, longer outputs, and perturbed inputs would clarify when each policy is suitable. The mixed PointMotionBench results also motivate examining why some error metrics improve while WorldTrack FDE and PWT do not.

🏭 Potential Industry Use Scenarios

The paper evaluates settings relevant to robotics and autonomous driving, where a high-level model produces plans that condition a separate action policy. In manipulation, its MolmoSpaces WAM consumes VLM features for the question, subtask, and 2D+3D trajectory while retaining camera and robot-state inputs. The LIBERO-family setup likewise conditions a PIv3 action head on generated subtasks or full plans. These are concrete examples of using structured plans as an interface between perception and action.

For driving, the evaluated policy reads prompt and plan states alongside ego history and vehicle state, then predicts a continuous trajectory. The paper reports latency measurements on embedded Jetson Thor, desktop-class DGX Spark, and server H100 hardware, making the work relevant to teams comparing on-device and server inference. These results suggest plausible applications where frequent spatial planning matters, but the evidence comes from the paper’s evaluated pipelines; real-world deployment would require validation of sensing, timing, and failure behavior in the intended operating environment.

💬 Critical Analysis

A strength of the work is its clear match between output structure and decoding unit. A box or trajectory has known boundaries and coordinate roles, so complete-primitive drafting avoids arbitrary fixed-window splits and can reduce the number of sequential passes. The paper also separates ordinary text from spatial output: it keeps target verification for text and uses structural acceptance for visual primitives. Shared-target comparisons in the trajectory tables help isolate the decoding change, while the downstream MolmoSpaces variants compare plan types and context features using the same WAM.

The evidence is broad across primitive types and application settings, but the reported results are not uniformly favorable across every metric. Dense-motion evaluation improves some ADE/FDE values while worsening WorldTrack FDE and lowering PWT across datasets. Box direct acceptance has a substantial Dense200 mean-F1 drop, and fallback changes both quality and speed. Such differences make per-task acceptance policy and metric direction central to interpreting the method.

The paper reports deployment latency and downstream success, but the supplied main-body evidence does not establish robustness under distribution shift or real-world operational failure rates. The approximately 25× dense-motion speed result is a single-rollout claim tied to its reported setup; end-to-end latency also depends on platform and task. Reproduction should therefore preserve the exact target, action model, data split, input views, output representation, and acceptance policy, and should report both spatial errors and downstream action outcomes. The paper’s figures and tables provide useful comparisons, but further independent validation would be needed before treating the results as general deployment guarantees.