RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions

📌 Highlights

RynnWorld-Latent uses continuous visual-transition descriptors as a shared conditioning interface for learning video dynamics from human, robot, and simulation data. Its goal is to avoid requiring native action spaces to match across embodiments.

The core model combines global action tokens with local, frame-aligned modulation of future video latents. After target-robot post-training, a separate encoder also allows conditioning on native robot commands.

The authors report improved prediction metrics on the public DreamDojo GR-1 sets and their held-out Marvin-WUJI and Astribot-S1 demonstration sets. They also report higher success rates with generated data on LIBERO, LIBERO-Plus, and evaluated real-robot tasks.

Evidence is bounded: real-robot prediction uses held-out demonstrations from represented tasks, action following is qualitative, and image metrics do not establish physical validity.

• Shared latent-action interface across human, robot, and simulation videos.
• Two data-generation routes: digital teleoperation and latent-action transfer under an edited initial frame.
• Reported gains apply to the evaluated benchmarks and robot settings; unseen-task and zero-shot transfer remain unestablished.

🎯 Introduction

The paper frames video world models as a way to learn visual dynamics from large video corpora, but argues that language conditioning is too coarse for precise manipulation. A task description such as “pick up the cup” does not determine motion timing, contact order, gripper state, or camera motion.

Conditioning on raw robot commands creates a different obstacle: embodiments vary in joint counts, coordinate conventions, control frequencies, and hardware limits. Human videos also often lack robot-specific controls, and simulation actions need not match a target robot's space.

The paper's objective is to use continuous latent actions distilled from visual transitions as a common interface, enabling joint pretraining without aligning native control spaces. It builds on RynnLAM and investigates latent descriptors as conditioning signals for visual dynamics.

🔬 Methodology

The model predicts future video latents from the first latent frame and a sequence of latent actions, as formalized in the conditional prediction distribution. The training corpus spans 23 datasets and includes human egocentric video, robot demonstrations, and simulation rollouts.

For each transition, frozen RynnLAM extracts a 608-dimensional descriptor with motion, global-action, and camera components. The video VAE maps an 81-frame chunk to 21 latent frames; the latent-action stride aligns 20 action descriptors with the future latent-frame transitions. Each descriptor is projected and passed through a residual MLP; resulting tokens enter joint self-attention.

A second pathway locally conditions the vision tokens for each corresponding future latent frame. A zero-initialized head produces modulation parameters, and a zero-initialized per-channel gate controls the action-token addition. The initial frame remains unmodulated. Training follows the paper's rectified-flow interpolation and velocity-prediction objective, with clean action conditioning and only future video latents denoised.

For target-robot adaptation, a robot-specific encoder maps synchronized commands into tokens used through the same conditioning pathway. Digital teleoperation generates video feedback and paired action-video data from operator commands. Latent-action synthesis edits a source video's first frame, reuses its latent actions to generate a rollout, then pairs it with temporally aligned retargeted pseudo-actions.

📊 Experiments

For public prediction benchmarks, the paper evaluates the DreamDojo GR-1 In-lab Eval, EgoDex Eval, and DreamDojo-HV Eval sets. Open-loop rollouts are conditioned on an initial frame and recorded native robot-action sequences; the last predicted frame autoregressively becomes the next conditioning frame. Higher PSNR and SSIM and lower LPIPS are better. RynnWorld-Latent reports the best listed values on all three sets, though baseline measurements are literature-reported and do not isolate the interface's contribution.

For the authors' real-robot prediction benchmarks, the model is evaluated on held-out episodes from represented tasks on Marvin-WUJI and Astribot-S1. Episodes are split before temporal chunking, with 5% reserved for testing and 95% used for post-training; this is a held-out-demonstration, not held-out-task, evaluation. RynnWorld-Latent reaches 21.86 PSNR, 0.80 SSIM, and 0.12 LPIPS on Marvin-WUJI, and 15.96, 0.60, and 0.34 respectively on Astribot-S1. Ablations show lower reported fidelity when removing latent-action pretraining, frame-aligned injection, or local modulation.

The paper also reports qualitative action following: distinct source latent-action sequences paired with the same initial observation produce rollouts that qualitatively resemble their corresponding source motions. This does not quantify physical accuracy or executable action tracking.

For policy learning, the paper reports success-rate percentages on LIBERO and LIBERO-Plus after synthetic-data mid-training, and closed-loop success rates for four real-world manipulation tasks: Plate Placement and Plate Wiping on Marvin-WUJI, Steak Serving and Plate Retrieval on Astribot-S1. Real-world settings use 35 trials per applicable task. The table preserves the reported values; gains are evidence for these evaluated settings, not a general guarantee.

fig:data_engine

Caption: Overview of the world-model data engine. Top: Scalable data editing. extracts latent actions from a source human video, while its first frame is edited to depict the target robot. Conditioned on the edited frame and the original latent actions, generates robot video rollouts. The generated observations are paired with pseudo-actions obtained through motion retargeting and inverse kinematics. Bottom: Digital teleoperation. Human control signals are retargeted to robot actions, which condition together with a reference scene. Generated frames provide visual feedback to the operator, while the recorded actions and generated observations form synthetic demonstrations without physical robot exec...

Why It Matters: It shows the two data-generation routes and makes clear where target-robot commands, transferred latent actions, generated frames, and retargeted pseudo-actions enter the pipelines.

fig:action_following

Caption: Action following under a shared initial observation. Each comparison uses latent-action sequences extracted from two different source videos. Both sequences are paired with the same initial frame to generate separate rollouts. Blue and green arrows associate source clips with their corresponding generated rollouts. The generated motion qualitatively follows the respective source sequence despite the shared initial scene.

Why It Matters: This is the paper's direct qualitative test of whether changing the supplied latent-action sequence changes generated motion while the initial observation is held fixed.

fig:Ego2Robot

Caption: RynnEdit. Two routes construct a robot-centric reference frame: explicit hand removal, retargeting, and robot rendering; or direct replacement of human hands with robotic counterparts using an image-editing model. The resulting first frame conditions subsequent world-model generation.

Why It Matters: It illustrates how the latent-action synthesis route constructs a target-robot initial observation from egocentric human video.

fig:vis-robot

Caption: Standardized Experimental Platforms. Two dual-arm embodiments used for real-world evaluation: the stationary Marvin-WUJI (dual 7-DOF arms with 20-DOF dexterous hands) and the mobile Astribot-S1 (dual 7-DOF arms with parallel grippers on a wheeled base).

Why It Matters: It identifies the two distinct robot platforms used for real-world prediction and policy evaluation.

🔮 Conclusion

The paper presents latent actions as a shared interface for cross-embodiment visual dynamics learning and demonstrates how a target-adapted model can generate synthetic robot data through teleoperation or source-motion transfer.

The reported experiments support improved video-prediction metrics on the evaluated public and robot benchmarks, qualitative action following, and improved downstream success rates in the listed simulation and real-world settings. The evidence supports these evaluated settings rather than a general claim of physical correctness or zero-shot deployment.

🛠️ Future Research Improvements

The evidence suggests testing whether the method transfers to held-out tasks and unadapted embodiments; the reported real-robot prediction split holds out episodes within tasks that are represented during post-training.

A useful next evaluation would measure physical consistency and action execution of generated trajectories directly, since image-based metrics and qualitative action-following examples do not establish either property.

Matched comparisons that control backbone, data, and post-training would help isolate the effect of the latent-action interface. The paper itself notes that its literature-reported DreamDojo baselines do not isolate these factors.

🏭 Potential Industry Use Scenarios

The demonstrated digital-teleoperation route could support collecting paired robot-action and visual data without executing each demonstration on a physical robot. Practical use would depend on validating the fidelity and safety of generated rollouts for the target robot.

The latent-action synthesis route could augment robot training data by transferring interaction motion from human videos into edited robot-centric scenes, with retargeted pseudo-actions paired to generated observations. Because embodiment, workspace, and contact changes can make the transfer infeasible, teams would need application-specific quality filtering and evaluation.

The paper also evaluates synthetic mid-training for VLA policies on LIBERO, LIBERO-Plus, and four real-world tasks. These results motivate testing generated data in comparable robot-learning workflows, while remaining specific to the reported setups.

💬 Critical Analysis

A strength is that the paper evaluates several distinct outcomes: public video-prediction benchmarks, held-out robot demonstrations on two platforms, qualitative action following, and downstream policy success. Its component ablations also report lower prediction metrics when removing latent-action pretraining, frame-aligned injection, or local modulation.

The scope of those results matters. The real-robot test split is within-task, public benchmark baselines are taken from literature, and policy success results cover the reported tasks and trials. Consequently, the comparisons do not prove broad transfer or isolate all sources of improvement.

Generated observations and retargeted pseudo-actions are produced independently, and the authors explicitly state that synchronization does not guarantee physical consistency. The model is therefore promising as a data-generation component, but the evidence calls for independent checks of executable actions, contact outcomes, and generalization before relying on synthetic episodes in deployment-critical training.