📌 Highlights
FWM's central contribution is an explicit interface between an executable combat engine and a generative video renderer. The proxy carries attempted actions, event timing, reactions, and persistent health in a common identity-aware representation, allowing the engine to change rules without retraining the renderer.
The strongest reported result is the full FWM interaction proxy's 98.48 joint fidelity with 99.78 / 99.24 / 99.06 movement, attack, and skill accuracy, 99.79 reaction accuracy, and 0.15 Health MAE. The most important caveat is scope: the experiments use a fixed-view, two-fighter Street Fighter II setting.
- Separates rule execution from visual generation.
- Binds character references to persistent fighters across side switches.
- Supports damage scaling, limited-hit immunity, lifesteal, and multi-stage skills without renderer retraining.
- Uses FightWorld Corpus with 112,618 samples, 52 skill effects, 187 character-appearance variants, and 15 arenas.
- Demonstrates large gains over adapted H3-WORLD and ReactiveGWM baselines in the reported setting.
🎯 Introduction
Interactive video world models can generate playable sequences from controller or text inputs, but combat requires more than plausible motion. A protected fighter may visibly react without losing health, lifesteal may restore the attacker's health, and appearance changes should not alter the underlying interaction. Systems that infer state transitions inside the video generator can retain learned action-consequence associations even after external rules change.
FWM addresses this mismatch by executing state transitions outside the renderer and compiling the results into a visual proxy. The paper's objective is to make action windows, hit reactions, persistent health, identity, and timing explicit enough that the same renderer can depict modified rules and changed character or arena appearances.
🔬 Methodology
The engine advances executable state with equation 1, producing both the next state and event records. Engine states are aligned to output frames, projected into visible fighter attributes, and combined with active action and reaction records. The compiler generates each proxy frame as a deterministic spatial-temporal encoding. Fighter boxes and HUD bars share identity colors; command stripes represent attempted attacks; skill markers identify casting windows; brief color inversions mark reactions; and bar length represents persistent health.
The renderer uses a pretrained Wan2.2-TI2V-5B video diffusion transformer with a shared VAE and patch embedding for video, proxy, and reference inputs. Proxy tokens share spatiotemporal rotary positions with video tokens, while identity slots are added to fighter regions in the initial video latent, proxy sequence, and corresponding character-reference tokens. Equation 3 describes the conditional video-generation process, with only future video latents denoised and only video tokens decoded.
FightWorld Corpus pairs Street Fighter II gameplay with interaction proxies and optional appearance references. A frozen video editor produces native, skill-effect, character-appearance, and scene edits, yielding 112,618 training samples over 52 skill effects, 187 character-appearance variants, and 15 arenas.
fig:corpus
Caption: FightWorld Corpus. Native gameplay is expanded with skill effects, character-appearance edits, and scene edits, yielding 112,618 training samples.
Why It Matters: The corpus construction connects native gameplay with controlled skill, character, and scene variation.
fig:state_rep
Caption: Complete-Information Encodings on the Same Interaction. Held-out rollout under matched executed records (tab:main). Green brackets mark HUD state errors: unexpected health loss, missing health loss, delayed update, incomplete update, and unexpected recovery.
Why It Matters: The figure illustrates why spatially aligned proxy cues can associate event timing and HUD updates more directly than symbolic text or state tokens.
equation 1
Meaning: The executable engine maps the current state and submitted actions to the next state and the events produced under rule configuration R.
equation 2
Meaning: The visible proxy state records each fighter's persistent identity, bounding box, facing, stance, coarse action label, and health at video frame t.
equation 3
Meaning: The renderer samples future video frames conditioned on the initial context, interaction proxy, appearance references, and text condition.
📊 Experiments
The evaluation uses two-fighter interactions in the fixed 2D view of Street Fighter II. The renderer is trained on FightWorld Corpus and compared with adapted H3-WORLD, ReactiveGWM v1, and ReactiveGWM v2. Move Accuracy, Attack Accuracy, and Skill Accuracy measure action control; Reaction measures visible hit responses; Health MAE measures normalized HUD-health error in percentage points; J requires all applicable action, entity, timing, reaction, and health checks to pass; and J_rule requires reaction and health checks to pass together.
Against adapted baselines, FWM reaches 99.78 / 99.24 / 99.06 action accuracy, 99.79 reaction accuracy, 0.15 Health MAE, 99.12 / 100.00 character/scene adherence, 0.60 character DINO similarity, and 0.72 scene DINO similarity. Under matched executed records, FWM reaches 98.48 joint fidelity versus 76.35 for Spatial Proxy + Text and 63.28 for Spatial Proxy + State Tokens. The ablations show complementary effects: removing damage-event cues reduces Reaction to 80.42, removing health bars raises Health MAE to 2.18, and removing the skill trigger reduces Skill accuracy to 79.28.
Rule interventions rerun the engine with changed damage magnitude, limited-hit immunity, or lifesteal while keeping references and text fixed. FWM reports J_rule values of 97.76 / 98.84 for reduced/amplified damage, 99.92 for limited-hit immunity, and 99.87 for lifesteal, with Health MAE values of 0.18 / 0.22, 0.21, and 0.19 respectively. The qualitative multi-stage demonstrations cover lingering ground damage, per-projectile hits, staggered area hits, action locks, and spatial confinement.
fig:dual_actions
Caption: Dual-Character Actions and Responses. Top rails show visible action summaries for both fighters. Per-frame badges indicate realized ( ) or unrealized ( ) fighter actions and responses; denotes strategy-level NPC text in ReactiveGWM v1; HUD state icons follow fig:state_rep.
Why It Matters: This comparison shows how baseline models miss fighter actions, reactions, or persistent health updates that FWM keeps aligned.
fig:skill_timing
Caption: Skill Timing and Event Consequences. Comparison on a projectile skill commanded at $f=20$; HUD state icons follow fig:state_rep.
Why It Matters: The projectile example tests whether health changes occur at the intended impact rather than at command time.
fig:skill_mechanisms
Caption: Skill-Level Rule Mechanisms Beyond Native Combat. Five combat mechanisms absent from native Street Fighter II/III, spanning lingering ground damage, per-projectile hits, staggered area hits, action locks, and spatial confinement (boundary locked).
Why It Matters: The figure demonstrates rule compositions that are not native to the source combat system.
tab:binding_control
Caption: Effect of Identity Binding. Variants share reference inputs and training budgets, differing only in identity binding. Adherence is a percentage. Values are ranked within each test subset.
| Test Subset | Binding | Adherence ↑ | DINO Similarity ↑ |
|---|---|---|---|
| Non-Switch | Off | 78.43 | 0.47 |
| Non-Switch | On | 99.16 | 0.61 |
| Side-Switch | Off | 53.68 | 0.35 |
| Side-Switch | On | 98.27 | 0.60 |
Why It Matters: Identity binding is especially important when fighters exchange screen sides, raising side-switch adherence from 53.68 to 98.27.
tab:main
Caption: Interaction Fidelity across Control Representations. We compare variants with different conditioning signals to assess the effects of interaction information and its encoding. Rates are percentages, and Health MAE is in percentage points of maximum health. Across tables, bold and underlined values mark the best and second-best results.
| Conditioning | Action ↑ Move / Attack / Skill | Reaction ↑ | Health MAE ↓ | J ↑ |
|---|---|---|---|---|
| Actions Only | 70.27 / 56.22 / 58.64 | 74.43 | 3.42 | 34.18 |
| Actions + Rule Text | 71.71 / 68.40 / 57.35 | 80.16 | 3.07 | 46.37 |
| Spatial Proxy Only | 99.19 / 92.27 / 46.58 | 78.28 | 2.31 | 33.46 |
| Spatial Proxy + State Tokens | 94.48 / 91.23 / 82.42 | 87.17 | 1.23 | 63.28 |
| Spatial Proxy + Text | 96.72 / 93.63 / 92.43 | 92.24 | 2.78 | 76.35 |
| FWM Interaction Proxy | 99.78 / 99.24 / 99.06 | 99.79 | 0.15 | 98.48 |
Why It Matters: The full interaction proxy improves joint correctness over action-only, text, spatial, and state-token representations under matched executed records.
tab:method_comparison
Caption: Comparison with Adapted Game World Models. Baselines are adapted to FightWorld Corpus. +Ref denotes the addition of our reference-conditioning branch. Rates are percentages, and Health MAE is in percentage points of maximum health. DINO measures cosine similarity. A dash indicates an inapplicable metric.
| Method | Action ↑ Move / Attack / Skill | Reaction ↑ | Health MAE ↓ | Adherence ↑ Character / Scene | Character DINO ↑ | Scene DINO ↑ |
|---|---|---|---|---|---|---|
| H3-WORLD | 43.85 / 62.61 / 47.58 | 48.25 | 8.75 | — | — | — |
| ReactiveGWM v1 | 70.15 / 78.98 / 48.55 | 35.56 | 4.59 | — | — | — |
| ReactiveGWM v2 | 50.24 / 59.70 / 16.67 | 49.21 | 4.65 | — | — | — |
| H3-WORLD + Ref. | 36.91 / 62.70 / 42.03 | 42.22 | 7.57 | 88.10 / 100.00 | 0.47 | 0.68 |
| ReactiveGWM v1 + Ref. | 63.16 / 85.53 / 53.86 | 33.97 | 4.79 | 89.29 / 100.00 | 0.50 | 0.64 |
| ReactiveGWM v2 + Ref. | 72.22 / 74.10 / 34.78 | 57.14 | 4.04 | 91.67 / 100.00 | 0.41 | 0.65 |
| FWM (Ours) | 99.78 / 99.24 / 99.06 | 99.79 | 0.15 | 99.12 / 100.00 | 0.60 | 0.72 |
Why It Matters: FWM leads the adapted baselines across action accuracy, reaction accuracy, health error, character adherence, and DINO similarity in the reported setting.
🔮 Conclusion
The paper concludes that FWM can connect executable gameplay rules to video generation through a visual interaction proxy. In the reported combat setting, the design improves action and reaction fidelity, persistent health consistency, and appearance control relative to adapted game world models. Identity binding is particularly effective for preserving character associations across side switches.
The practical takeaway is architectural: an authoritative engine can remain responsible for rule semantics while a generative renderer handles detailed motion, textures, and effects. The main-body evidence supports this division for the tested fixed-view combat domain, but does not yet establish broad game-world generality or operational real-time performance.
fig:appearance_ref
Caption: Character and Scene Reference Conditioning. Interaction rollouts across four character appearances (under default and side-switched stances) and two arena references.
Why It Matters: The figure visualizes appearance customization while preserving the same interaction records across characters and arenas.
🛠️ Future Research Improvements
The main body does not provide a detailed future-work agenda, so the following directions follow directly from the evaluated scope. The first priority is testing whether the proxy and identity-binding design transfers to 3D environments, moving cameras, more than two fighters, and game engines with substantially different state representations.
A second priority is operational evaluation: report generation latency, memory use, rollout length, failure recovery, and temporal consistency under interactive workloads. Additional human or task-based evaluations would complement VLM, DINO, and HUD-based metrics, especially for subtle identity drift and visually plausible but semantically incorrect reactions.
🏭 Potential Industry Use Scenarios
The architecture could support configurable combat prototypes in which designers edit damage, immunity, lifesteal, status, or multi-stage skill rules in an executable engine and immediately render alternative visual outcomes through the same trained renderer. This is directly aligned with the paper's demonstrated rule interventions, but deployment suitability would require latency and robustness measurements not reported in the main body.
Other plausible uses include rapid character and arena concept visualization, interactive game-design demonstrations, and automated testing of whether generated combat footage preserves prescribed action timing and HUD state. These scenarios are extensions of the demonstrated capabilities rather than results established outside the fixed two-fighter benchmark.
💬 Critical Analysis
FWM's strongest design choice is the explicit separation of attempted actions from executed consequences. The controlled comparisons and ablations support this mechanism: spatial alignment improves action and event association, damage-event cues primarily affect reactions, health bars primarily affect persistent state, and skill triggers are important for skill accuracy. The rule-intervention results also test a meaningful property that symbolic conditioning alone struggles with: a visible reaction can coexist with immunity, and one hit can produce coupled health changes through lifesteal.
The main uncertainty is external validity. The benchmark is a controlled two-fighter, fixed-view Street Fighter II environment, and the corpus includes systematic visual and skill edits rather than evidence from independent game engines. The evaluation uses automated judges and proxy-derived health checks, which are useful for controlled measurement but may not capture all perceptual or gameplay failures. Reproduction should therefore verify long-horizon stability, unseen rule compositions, identity persistence during rapid contact, and deployment cost before treating the reported fidelity as evidence of a general-purpose game world model.