Trainspotting: Learning Object-Centric 3D Reconstruction from Found Photos

📌 Highlights

TRAINSPOTTING targets a specific failure mode in feedforward 3D reconstruction: object photos collected independently across time and context. Its 12,068 objects and 311,886 photos are divided into fixed, roaming, and instance-pooled regimes, and the authors reconstruct poses, MVS depth, and masks to create supervision.

Fine-tuning VGGT-Ω produces SPOTTER, which improves every reported metric on held-out TRAINSPOTTING objects and substantially improves NAVI in-the-wild camera pose and completeness. The strongest evidence is the jump in AUC@30 from 25.0 to 81.2 on TRAINSPOTTING and from 18.8 to 76.0 on NAVI in-the-wild.

The main caveat is that TRAINSPOTTING labels and geometry are automatically constructed and therefore are not independent of the reconstruction system used to create them.

• Core claim: capture diversity is a missing training signal for object-centric reconstruction.
• Strongest held-out result: SPOTTER reaches 81.2 AUC@30 versus 25.0 for VGGT-Ω.
• Independent transfer: NAVI in-the-wild AUC@30 rises from 18.8 to 76.0.
• Most important caveat: the primary dataset uses pseudo-ground truth from an automated reconstruction pipeline.

🎯 Introduction

Most object-level 3D reconstruction datasets use coordinated videos or capture sessions in which viewpoints vary while the object and environment remain nearly fixed. Found photos in public archives follow a different distribution: images may be taken by different people, with different cameras, over years, in different environments, and sometimes of different physical instances sharing one design.

This setting breaks assumptions behind repeatable correspondences and rigid-scene structure from motion. The paper's Buick Riviera example shows how VGGT-Ω can scatter camera poses and geometry when sixteen images depict multiple instances of one car design. The paper therefore asks whether a large object-centric web dataset, reconstructed with robust matching and geometric checks, can improve feedforward models on this distribution shift.

The proposed objective is both a dataset contribution and a training intervention: build object-level supervision at scale, evaluate recent models under changing capture conditions, and test whether fine-tuning on those conditions transfers to an independent in-the-wild benchmark without materially harming conventional reconstruction performance.

🔬 Methodology

TRAINSPOTTING begins with Wikimedia Commons categories and Wikidata-linked classes. A language model filters candidate categories for rigid physical objects or designs and assigns approximate regime labels. The final dataset contains 4,338 fixed objects, 2,869 roaming objects, and 4,861 instance-pooled objects; the median capture span is 12 years, 90% of objects span at least one year, and a median object has photos from 7 photographers and 6 camera models.

For each collection, SAM3 performs object gating and retains images whose largest detected object box covers at least 10% of the image. Images are embedded with MASt3R-SfM retrieval and paired with the 20 nearest images. RoMa v2 supplies dense correspondences, which are retained only when they pass the confidence, cycle, and epipolar checks in Equation 1. The surviving matches become multi-view tracks for incremental COLMAP reconstruction.

View selection removes weakly connected or misregistered views using track length and a covisibility graph. Cas-DiffMVS estimates depth, MoGe-2 verifies it, and SAM3 supplies the object mask selected by overlap with verified depth. The final quality checks remove erroneous objects, implausible focal lengths, and images where the object is not dominant.

SPOTTER fine-tunes the released 1B-parameter VGGT-Ω checkpoint using connected samples of 2–24 photos. It updates the camera head and transformer attention blocks, trains with the camera, depth, and point losses in Equations 2–5, discards the largest 2% of residuals in each batch, and replays 15% of samples from BlendedMVS, Hypersim, and uCO3D.

• Dataset: 12,068 objects and 311,886 photos from Wikimedia Commons.
• Regimes: fixed, roaming, and instance-pooled.
• Reconstruction: SAM3, MASt3R-SfM retrieval, RoMa v2, MAGSAC++, COLMAP, Cas-DiffMVS, MoGe-2, and SAM3 masks.
• Training: 7,500 steps, 768 images per step, AdamW, peak learning rate 10−5, and four NVIDIA B200 GPUs for about 12 hours.

📊 Experiments

The main held-out evaluation uses one connected 4-view sample from each of 1,000 held-out TRAINSPOTTING objects and an additional connected 16-view sample when available, yielding 1,635 samples. External evaluation uses one fixed-seed 8-view sample per scene from NAVI in-the-wild, NAVI, 7-Scenes, NRGBD, DTU, and ETH3D. Camera pose is measured with pairwise relative-rotation and translation-direction AUC@5 and AUC@30; depth uses AbsRel and δ1.03 after one median scale per sample; point maps use similarity alignment refined by ICP.

On TRAINSPOTTING, SPOTTER improves over VGGT-Ω from 1.84 to 1.22 accuracy error, 12.68 to 1.15 completeness error, 0.201 to 0.071 AbsRel, 28.1 to 53.3 δ1.03, 7.2 to 44.2 AUC@5, and 25.0 to 81.2 AUC@30. The view-count and regime results show that VGGT-Ω is particularly weak on roaming and pooled objects, while SPOTTER reaches 79.9, 83.2, 79.0, 81.0, and 82.6 AUC@30 for 4 views, 16 views, fixed, roaming, and pooled settings respectively.

NAVI in-the-wild provides the strongest independent transfer test. Relative to VGGT-Ω, SPOTTER raises camera AUC@30 from 18.8 to 76.0, improves point-map completeness from 20.4 to 1.9 mm, and improves depth inlier ratio from 29.7 to 43.1. On NAVI single-session captures and conventional benchmarks—7-Scenes, NRGBD, DTU, and ETH3D—the changes are comparatively limited, consistent with the reported goal of retaining existing capabilities.

The training-data ablation compares all 11,068 training objects, a random 20% subset of 2,193 objects, and regime-specific subsets. The full set reaches 80.3 AUC@30 on all held-out objects and 74.6 on NAVI in-the-wild, while fixed-only training reaches 68.5 and 63.4. Training only on fixed objects is also much weaker on roaming and pooled test groups, supporting the authors' claim that cross-environment variation, rather than scale alone, is the crucial training signal.

• Datasets and benchmarks: TRAINSPOTTING, NAVI in-the-wild, NAVI, 7-Scenes, NRGBD, DTU, and ETH3D.
• Training replay sources: BlendedMVS, Hypersim, and uCO3D.
• TRAINSPOTTING camera AUC@30: VGGT-Ω 25.0; SPOTTER 81.2.
• NAVI in-the-wild camera AUC@30: VGGT-Ω 18.8; SPOTTER 76.0.
• Conventional benchmark changes remain small: at most 1.1 AUC@30 points and 3.2 δ1.03 points relative to VGGT-Ω, as reported by the authors.

fig:pipeline

Caption: Reconstruction pipeline for the Flying Scotsman. (a) Object gating retains 570 of 727 photos, with red borders marking excluded photos. (b) A retrieved pair is shown with candidate correspondences. Green correspondences pass the confidence, cycle, and epipolar checks, while dashed red correspondences fail. (c) The structure-from-motion reconstruction registers 496 images, merged from 14 partial reconstructions. View selection keeps 481 of them for multi-view stereo. (d) The same pair shows MVS depth before and after MoGe-2 filtering. (e) Four of the 370 photos in the dataset show MVS depth in color and the selected mask in teal over the image.

Why It Matters: This figure makes the data-generation chain concrete, showing how noisy web photos are filtered, registered, depth-verified, and converted into object-centric supervision.

fig:gallery

Caption: Examples of capture variation. Each column displays photos of example objects, grouped into the three regimes (fixed, roaming, and instance-pooled) and labeled with their year and camera. Fixed objects stay at one site, roaming objects are single instances that move between sites, and instance-pooled objects are different physical instances of one shared design.

Why It Matters: The three regimes define the distribution shift the model must learn: environmental change for roaming objects and physical-instance variation for shared designs.

fig:failure

Caption: VGGT-$ $ fails on the Buick Riviera. We visualize the camera poses and point map predicted by VGGT-$ $ from sixteen photos of the Buick Riviera (multiple physical instances of the same car design). The views are misregistered and the car is reconstructed as scattered fragments. Figure~fig:qualitative shows the reconstruction of our fine-tuned model from the same photos.

Why It Matters: The failure case illustrates why conventional feedforward reconstruction can break when photos depict multiple physical instances of a design.

Figure ure

Caption: Figure 6. Qualitative comparison. We visualize ground truth and predicted camera poses and point maps from π3 [52], Depth Anything 3 [25], VGGT-Ω[49], and SPOTTER on held out objects from TRAINSPOTTING and NAVI in-the-wild [19]. We use 4 or 16 input views for TRAINSPOTTING and 8 for NAVI in-the-wild. Only the first 4 input views are shown. Triangular markers indicate the directions of the cameras outside of the frame. The baselines misregister the input views and produce duplicated or fragmented geometry, whereas SPOTTER recovers a single consistent object and accurate camera poses.

Why It Matters: The qualitative comparison shows the central behavioral change after fine-tuning: SPOTTER produces one coherent object and consistent camera poses where baselines produce fragments or duplicated geometry.

🔮 Conclusion

The paper argues that current feedforward reconstruction models are not sufficiently exposed to the way object photographs occur in public archives. TRAINSPOTTING turns that distribution into a large object-centric resource, and SPOTTER demonstrates that fine-tuning on changing environments and multiple physical instances can improve pose, depth, and geometry.

The strongest evidence is consistent across held-out TRAINSPOTTING objects and the independent NAVI in-the-wild benchmark, while performance on NAVI single-session data and conventional benchmarks remains broadly stable. The authors conclude that the missing ingredient is variation in how an object is photographed, not simply more examples of coordinated captures.

The resulting reconstructions may also support analysis of how objects and their surroundings change over time, although that use depends on the reliability of automatically recovered geometry and identity assignments.

🛠️ Future Research Improvements

A natural next step is to validate more of TRAINSPOTTING against manually checked camera poses, depth, masks, and object identity, separating model improvement from possible artifacts introduced by the pseudo-ground-truth generator.

The regime labels could be refined with stronger temporal, geographic, and instance-level metadata. This would allow experiments that distinguish environmental change, object modification, and cross-instance shape variation rather than grouping them under broad approximate labels.

Future evaluations should expand beyond Wikimedia Commons and NAVI, test more severe occlusion and sparsity, and compare full fine-tuning with parameter-efficient adaptation or smaller reconstruction backbones. The current paper establishes transfer for one 1B-parameter model and a defined set of benchmarks, leaving these efficiency and robustness questions open.

• Measure label and reconstruction uncertainty rather than treating all pseudo-ground-truth samples equally.
• Separate object aging or modification from camera and environment variation.
• Test cross-archive and cross-domain transfer.
• Report compute-quality tradeoffs for smaller or parameter-efficient models.

🏭 Potential Industry Use Scenarios

Cultural-heritage and transportation archives could use the approach to consolidate photographs of locomotives, aircraft, ships, statues, monuments, and other assets into coherent 3D references, especially when no coordinated capture session exists.

Asset search and cataloging systems could benefit from object-centric reconstruction that is robust to changing backgrounds and viewpoints. For example, a museum, infrastructure operator, or manufacturer could organize historical photographs around a persistent object or design and use the recovered geometry for inspection or visualization.

Mobile-asset monitoring is a plausible application for objects that move between sites, while instance-pooled designs could support product-family reconstruction from heterogeneous images. These scenarios require additional validation of identity consistency, licensing, geometry accuracy, and performance under the target organization's image distribution.

• Historical and cultural-heritage 3D archiving.
• Fleet, rolling-stock, ship, and aircraft documentation.
• Product-family visualization from heterogeneous photographs.
• Object-centric image search and asset cataloging.
• Longitudinal analysis of object and environment changes.

💬 Critical Analysis

The paper's central strength is that it identifies and measures a concrete distribution mismatch rather than treating all multi-view data as equivalent. The regime breakdown, independent NAVI in-the-wild evaluation, and fixed-only ablation jointly support the interpretation that environmental and instance variation are causally relevant training signals.

The most important methodological concern is dependence between dataset supervision and the reconstruction system. TRAINSPOTTING uses automated gating, matching, COLMAP registration, MVS, monocular filtering, and masks; failures in those stages may bias both the fine-tuning targets and the held-out evaluation. The NAVI results reduce this concern but do not eliminate it because the independent in-the-wild sample count is small.

The metrics also require careful reading. Similarity alignment can make misregistered predictions appear relatively accurate on surface points while leaving much of the object uncovered, which is why the authors emphasize completeness. A builder should therefore avoid optimizing only point accuracy or depth error and should inspect camera consistency and coverage directly.

Overall, the evidence is strong for adapting VGGT-Ω to object-centric, cross-environment imagery, but the paper does not yet establish universal robustness, low-cost adaptation, or independence from automated reconstruction biases.

• Strength: clear problem decomposition into fixed, roaming, and instance-pooled regimes.
• Strength: improvements transfer to NAVI in-the-wild without major conventional-benchmark degradation.
• Caveat: primary supervision and evaluation are partly coupled to the authors' reconstruction process.
• Caveat: small independent in-the-wild evaluation and approximate regime labels limit generality.
• Open question: whether the same gains hold for smaller models, other archives, and substantially noisier imagery.