📌 Highlights
The paper asks whether latent prediction error is a reliable measure of physical surprise. Its central finding is that physical information may be present in representations, and action mismatch may increase error, while the native score still responds to appearance and static content.
In the controlled support experiment, impossible-minus-possible error changes sign between solid support and a gap. A threshold fitted on separate clean calibration configurations reaches 53.9% balanced accuracy and falsely flags about 69% of lawful clips.
- V-JEPA2-G reaches 57.9% IntPhys2 physical-label accuracy on held-out scenes and 97.7% appearance-label accuracy.
- Repeating a single frame raises mean error in the evaluated Physion++ diagnostic; persistent hue shifts and pixel darkening lower it.
- Opposite actions raise error across LeWM, DINO-WM, and JEPA-WM when the observed outcome is fixed.
- The evidence is limited to the evaluated models and controlled settings; real-world and material generalization remain untested.
🎯 Introduction
Video world-model prediction errors are used to assess physical plausibility and identify unexpected events. Yet the score measures prediction difficulty in a chosen representation, so it can also respond to appearance, static content, motion, or ordinary model error. The paper's objective is to distinguish what frozen representations encode from what changes a model's native error and whether that score supports physical-validity decisions.
The study probes IntPhys2, which pairs physically possible and impossible events, and compares latent prediction features from V-JEPA2-G with masked-pixel reconstruction features from VideoMAE2-G and simple visual statistics. It then examines layerwise appearance and physical geometry and tests controlled interventions in action-conditioned models, Physion++, and Kubric.
🔬 Methodology
The evaluation framework separates representation evidence from score evidence. Linear probes test whether physical labels are readable from frozen features. Geometry measures appearance displacement relative to unrelated-video distances, directional alignment across scenes, and overlap with physical-pair changes. Controlled score tests compare correct and mismatched actions for the same outcome, appearance or static-content changes at fixed physical state, possible and impossible events across conditions, and fixed-threshold transfer to new scenes and appearances.
For Kubric, the study crosses solid support versus a gap, stopping versus falling, and original, persistent-hue, or abrupt-hue appearances, yielding 256 configurations and 3,072 clips. V-JEPA2-G predicts the remaining 36 frames after observing the first 12 of 48; error is mean absolute difference from normalized EMA-target tokens. Separate experiments evaluate V-JEPA2-L on Physion++ Mass, action-conditioned models on TwoRoom and PushT, and IntPhys2 probes. Bootstrap units are configurations, matched pairs within scenarios, scenes, or action-test episodes; native error magnitudes are compared only within each model.
The score being audited is the distance between predicted and target features, conditioned on context and, when applicable, action.
display equation 1
Meaning: Defines the native prediction-error score as the distance between the action-conditioned prediction from context features and the target features.
📊 Experiments
The central datasets and tasks are IntPhys2 for possible-versus-impossible events, Kubric for controlled support and trajectory interventions, Physion++ Mass for static-frame and appearance diagnostics, and TwoRoom and PushT for action-conditioned prediction. IntPhys2 comparisons include VideoMAE2-G and PixelSummary. DINO-WM and JEPA-WM are evaluated on PushT with a shared frozen DINOv2 encoder, varying the predictor. Uncertainty uses bootstrap intervals over the respective independent experimental units.
On held-out IntPhys2 scenes, V-JEPA2-G achieves 57.9% physical classification accuracy [54.2, 61.8] and 97.7% appearance classification accuracy [95.7, 99.3]. VideoMAE2-G reaches 49.2% and 95.3%; PixelSummary reaches 50.0% and 79.1%. These probe results show accessible representation information, not native-error detection performance.
In Kubric, impossible-minus-possible error reverses sign between solid support and a gap. A threshold calibrated on 24 separate clean configurations gives 53.9% balanced accuracy [52.1, 55.8] on clean evaluation clips, but performance is 80.7% over gaps versus 27.1% on solid support; about 69% of lawful clips are falsely flagged. In Physion++ Mass, repeating a single frame produces higher mean error than the original clip, while persistent hue and darkening reduce error on average. For action sensitivity, opposite-minus-lawful effects are positive for every listed model and task.
tab:intphys-readouts
Caption: IntPhys2 classifiers evaluated on held-out scenes. Top rows show mean scene accuracy (percent correct); bottom rows subtract the comparator's accuracy from V-JEPA2-G's (percentage points). Physics uses all four appearances or clean clips only. Brackets are 95\% global scene-bootstrap intervals with classifier predictions held fixed.
| Feature family or contrast | Appearance classification | Physics classification | Clean-only physics |
|---|---|---|---|
| V-JEPA2-G | 97.7% [95.7, 99.3] | 57.9% [54.2, 61.8] | 54.8% [51.3, 58.8] |
| VideoMAE2-G | 95.3% [92.8, 97.6] | 49.2% [47.3, 50.9] | 49.6% [46.5, 52.2] |
| PixelSummary | 79.1% [73.7, 84.5] | 50.0% [48.9, 51.0] | 51.8% [48.7, 54.8] |
| V-JEPA2-G − VideoMAE2-G | 2.4 pp [−0.1, 5.0] | 8.7 pp [4.4, 13.3] | 5.3 pp [0.4, 10.1] |
| V-JEPA2-G − PixelSummary | 18.6 pp [13.5, 24.1] | 7.9 pp [4.1, 11.8] | 3.1 pp [−1.3, 8.3] |
Why It Matters: Reports held-out-scene readout performance and shows that physical information is modestly accessible, distinct from the much stronger appearance readout.
tab:physion-appearance
Caption: V-JEPA2-L appearance effects on 96 matched Physion++ pairs, in native error units. Each entry subtracts the second condition from the first: negative means lower error, not a negative error score. Brackets are 99.17\% pair-bootstrap intervals, Bonferroni-adjusted for six comparisons. Persistent changes affect the whole clip; target-only changes affect its final four frames.
| Contrast | Hue | Pixel darkening |
|---|---|---|
| Persistent − original | −0.00312 [−0.00385, −0.00236] | −0.00270 [−0.00337, −0.00203] |
| Target-only − original | −0.00069 [−0.00168, +0.00021] | +0.00059 [−0.00038, +0.00158] |
| Persistent − target-only | −0.00243 [−0.00337, −0.00134] | −0.00329 [−0.00415, −0.00242] |
Why It Matters: Shows persistent color and brightness edits lower native error on average, while target-only effects relative to original have intervals that include zero.
tab:native-error-summary
Caption: Same-target action effects: opposite minus lawful action at full strength, over 256 episodes per task or scorer. Brackets are 95\% episode-bootstrap intervals. Each model uses its native latent MSE; magnitudes are not comparable across models.
| Model/task | Opposite − lawful |
|---|---|
| LeWM TwoRoom | 1.884 [1.821, 1.946] |
| LeWM PushT | 0.3834 [0.3456, 0.4242] |
| DINO-WM PushT | 0.00364 [0.00304, 0.00428] |
| JEPA-WM PushT | 0.00351 [0.00293, 0.00411] |
Why It Matters: Confirms that all evaluated action-conditioned models respond to opposite actions for the same observed outcome, with model-specific native score scales.
fig:final256
Caption: V-JEPA2-G on 256 configurations. Left: impossible minus possible native error, with 98.75\% configuration-bootstrap intervals adjusted for four means. The sign reverses between solid support and a gap. Right: balanced accuracy under a single threshold fitted on separate calibration scenes, with descriptive 95\% configuration-bootstrap intervals. The dashed line marks 50\%.
Why It Matters: Shows that the error ordering reverses with support context and that a single transferred threshold has limited balanced accuracy.
fig:layerwise-structure
Caption: Representation geometry on 57 IntPhys2 scenes. A: displacement relative to the median unrelated-video distance (log axis). B: directional coherence (mean resultant length, MRL) above shuffled correspondences. C: fraction of squared physical-pair displacement captured by a four-dimensional appearance subspace fitted on other scenes, with scene-bootstrap intervals and an equal-rank random reference.
Why It Matters: Visualizes how appearance displacement, directional coherence, and overlap with physical changes vary across representation depth.
fig:lewm-controls
Caption: LeWorldModel error changes. Action points compare incorrect and correct actions for the same outcome. Hue points compare changed color with the unchanged static target. The correct-action difference is zero and omitted.
Why It Matters: Compares action-mismatch and hue-change responses while controlling the relevant outcome or static target.
🔮 Conclusion
The paper's results distinguish physical information in a representation from physical specificity in its prediction-error score. V-JEPA2-G exposes modest physical-label information on IntPhys2, and action-conditioned models respond to mismatched actions, but appearance and static-content interventions also change error.
In the crossed-support test, prediction error follows trajectory identity strongly enough that physical-violation ordering reverses across support settings. The authors conclude that prediction errors can serve as physical-surprise measures only when controlled tests establish what the score responds to.
🛠️ Future Research Improvements
The authors identify broader controlled data as a needed direction: the support experiment uses one procedural family and restricted cameras, colors, and lighting. Testing additional materials and training diversity could establish whether appearance sensitivity changes under richer conditions.
The paper also leaves open whether removing or controlling appearance directions changes prediction-error specificity. Its tested linear geometry models do not establish a mechanism, so interventions on representation components and broader model comparisons would be useful follow-ups.
Extend evaluation to real-world scenes and application-specific decision costs. The paper does not specify acceptable false-alarm or missed-violation rates for a deployment.
🏭 Potential Industry Use Scenarios
Video anomaly detection and content review could use the six checks to test whether a model's score tracks the target class of physical violations across appearance and motion conditions. The paper does not establish deployment performance on real footage, so this is an evaluation approach rather than evidence of production readiness.
Robotics and simulation systems could audit action-conditioned world-model errors before using them as planning rewards or safety signals. The TwoRoom and PushT results show action sensitivity, but the paper does not evaluate downstream planning or real-robot success.
💬 Critical Analysis
A strength is the separation of representation readout, representation geometry, predictive sensitivity, and decision-threshold behavior. Matched interventions expose a practical failure mode that a representation probe alone would miss: an error score can respond to the physical action while also reacting to color or static content.
The strongest evidence against a general physical-surprise interpretation is the controlled Kubric sign reversal and poor threshold transfer. The IntPhys2 probe supports only modest accessible physical information, while its probe intervals condition on fitted predictions. The geometry-to-error regression adds little beyond experimental conditions and visual statistics, and is limited to the tested linear models.
These experiments justify caution for the evaluated models and settings, not a universal claim that latent prediction error cannot measure physical surprise. Model architecture and training differ, native scales are incomparable, and real-world generalization and task-specific operating requirements remain unresolved.