FlowMoDE: Coarse-to-fine Flow Matching for Detail-aware Sim-to-Real Monocular Depth Estimation

📌 Highlights

FlowMoDE addresses monocular depth estimation with a coarse-to-fine flow-matching architecture that combines global semantic context with fine-grained structural restoration.

The model is trained on 54K Hypersim frames and evaluated zero-shot on NYUv2, KITTI, ETH3D, ScanNet, and DIODE. The paper reports an average relative-error reduction of 14.1% against state-of-the-art works and a 9.9% reduction in Chamfer distance for point-cloud reconstruction.

The strongest evidence is the benchmark table: FlowMoDE reports an average rank of 2.2 across 10 metrics, compared with 3.7 for UniDepth v2 and 6.1 for PPD.

The main caveats are occasional flying pixels, blurred distant edges, and the absence of quantitative robotic-scene evaluation.

• Two-step flow-matching inference.
• Semantic conditioning from frozen DINOv2-B.
• Synthetic-only training with real-world zero-shot evaluation.
• Lower generative-model compute than DepthMaster and PPD.

🎯 Introduction

Monocular depth estimation supports scene reconstruction, robotic manipulation, and autonomous navigation. The paper identifies a complementary weakness in the two dominant paradigms: discriminative models often lose boundary and high-frequency detail after large-patch tokenization, while generative models can recover structure but incur high computational cost or lose global scene context.

FlowMoDE is designed to address both issues. It uses semantic features from a pretrained discriminative encoder to preserve global context, then applies a lightweight coarse decoder followed by a fine-grained decoder with a small number of Transformer blocks.

The paper's objective is therefore not simply lower error, but efficient sim-to-real depth prediction that preserves delicate structures and transfers from one synthetic indoor dataset to diverse real-world and robotic scenes.

🔬 Methodology

Flow matching learns a time-dependent velocity field that transports samples from a simple source distribution to the data distribution. The model uses conditional flow matching with a noisy depth representation and an input RGB image. The semantic encoder produces a condition map, while the timestep is jointly embedded with that semantic condition.

The architecture has three decoder stages. The Semantic Decoder processes projected DINOv2-B features. The Coarse Decoder operates on 16×16 image-depth patches and uses four conditioned Transformer blocks. Its output is upsampled to 8×8 patches, after which the Fine-grained Decoder uses four more conditioned Transformer blocks to recover local structure before the final feature map is unpatched into a velocity map.

During training, the target velocity is the difference between Gaussian noise and ground-truth depth, optimized with a masked MSE over valid depth pixels. During inference, the shared network is applied twice with half-step updates, producing the final clean depth estimate after two velocity predictions.

• Semantic condition dimension: 384.
• Semantic, coarse, and fine-grained decoder dimensions: 384, 384, and 768.
• Training uses normalized log-depth and Gaussian depth noise.
• Inference uses two shared-network flow steps.

pipe

Caption: Pipeline of the proposed method.

Why It Matters: The pipeline shows how semantic encoding, coarse decoding, fine decoding, modulation, and two-step inference are connected.

📊 Experiments

Training uses Hypersim, a synthetic indoor dataset with 54K frames and corresponding depth maps. Evaluation follows Marigold on five real-world datasets: NYUv2, KITTI, ETH3D, ScanNet, and DIODE. Inputs are resized with a long edge of 768, and the reported metrics are absolute relative error AR↓ and threshold accuracy δ1↑.

On the depth benchmarks, FlowMoDE reports AR/δ1 values of 4.0/98.1 on NYUv2, 6.2/95.9 on KITTI, 3.9/98.8 on ETH3D, 4.3/97.6 on ScanNet, and 18.8/80.4 on DIODE. Its average rank is 2.2, compared with 3.7 for UniDepth v2, 6.1 for PPD, and 6.9 for DepthMaster.

For point-cloud reconstruction on NYUv2 and ETH3D, the paper evaluates near-field and far-field geometry using Acc↓ and Chamfer distance CD↓. FlowMoDE obtains mean Acc 179.5 and mean CD 199.7, compared with 210.9 and 221.6 for DepthMaster and 506.5 and 378.4 for PPD.

Ablations show that removing semantic conditioning is especially damaging, with AR/δ1 degrading to 20.6/65.5 on NYUv2 and 26.0/56.9 on KITTI. Two flow steps are best overall: increasing from two to four, six, or eight steps does not improve the reported metrics. The efficiency table reports 3140.1 GFLOPs, 53.4 ms/image, 1.87 GB memory, and 398.7M parameters for FlowMoDE.

• Robotic scenes include Baxter, Franka, Kuka, and Da Vinci embodiments.
• Robotic evaluation is qualitative and compares depth and point-cloud reconstructions against PPD, MoGe-2, and UniDepth v2.
• The paper reports better reconstruction of robot bodies, cables, faucets, windows, and other delicate structures.

qual

Caption: Qualitative comparison on indoor examples (first four rows) and outdoor examples (last four rows). Figures in the second row of each sample are GT and corresponding error maps based on Abs Rel.

Why It Matters: The figure provides qualitative indoor and outdoor evidence for boundary and fine-detail preservation.

recon

Caption: Comparison of point cloud reconstruction based on estimated depth maps.

Why It Matters: The reconstructed point clouds connect image-space depth accuracy to downstream geometric fidelity.

rob_dp

Caption: Qualitative comparison of monocular depth estimation in robotic scenarios.

Why It Matters: The robotic examples test zero-shot transfer to scenes containing multiple robot embodiments and delicate components.

rob

Caption: Qualitative comparison of point cloud reconstruction based on estimated depth maps in diverse cross-embodiment robotic scenes.

Why It Matters: The figure illustrates the reported reduction of flying points between robot bodies and surrounding objects.

🔮 Conclusion

The paper presents FlowMoDE as an efficient flow-matching solution for sim-to-real monocular depth estimation. Its central design combines semantic guidance, coarse-to-fine decoding, and two-step inference to preserve fine structures while avoiding the cost of multi-step diffusion.

Across five real-world benchmarks, point-cloud reconstruction tests, and qualitative cross-embodiment robotic scenes, the reported evidence supports strong zero-shot generalization from synthetic indoor training. The authors nevertheless identify flying pixels and distant-region blur as unresolved failure modes.

🛠️ Future Research Improvements

The clearest next research direction is to address the limitations explicitly identified by the authors: patch-induced flying pixels and blurred distant edges. Boundary-aware supervision, higher-resolution tokenization, or non-patch-wise refinement would be natural hypotheses to test, although the supplied main body does not evaluate them.

The evaluation could also be strengthened with quantitative robotic-scene benchmarks, broader outdoor and long-range training data, and tests that separate depth accuracy from downstream manipulation or navigation performance.

• Quantify robotic-scene performance.
• Evaluate distant-object and thin-structure error separately.
• Test higher-resolution or adaptive patch representations.
• Measure downstream robot-task performance.

🏭 Potential Industry Use Scenarios

FlowMoDE is most directly relevant to robotic perception pipelines that need depth and geometric structure from a single RGB camera, including scene reconstruction, manipulation, and navigation. The cross-embodiment examples suggest potential use in environments containing Baxter, Franka, Kuka, or Da Vinci robots.

The synthetic-only training setup could be useful where collecting dense real depth labels is difficult, including specialized environments such as surgical scenes. Deployment should still verify long-range edge quality, flying-pixel behavior, latency on the target hardware, and the relationship between depth metrics and task success.

• RGB-only robot scene understanding.
• Point-cloud generation for manipulation and reconstruction.
• Zero-shot perception in specialized environments with limited labeled data.
• Efficient generative depth estimation where multi-step diffusion is too slow.

💬 Critical Analysis

The strongest aspect of FlowMoDE is the alignment between its architectural claim and ablation evidence. Semantic removal causes severe degradation, the complete decoder arrangement performs best among the tested configurations, and two-step inference is sufficient in the reported experiments. The point-cloud table further shows that the gains are not limited to pixel-space depth metrics.

The comparison is also nuanced by computational cost. FlowMoDE uses 3140.1 GFLOPs, 53.4 ms/image, 1.87 GB memory, and 398.7M parameters: it is far cheaper than DepthMaster and PPD in the supplied table, but slower and larger than UniDepth v2. Thus, the practical advantage is primarily relative to generative depth models, not across every model family.

The main remaining question is robustness outside the tested distribution. The method is trained on near-field indoor synthetic data, while several conclusions concern outdoor and robotic scenes. The paper provides useful qualitative evidence for those settings, but quantitative robotic evaluation, detailed failure-rate analysis, and broader long-range testing would be needed before treating the method as deployment-ready.