Mask2Real-WM: Segmentation Masks as a Sim-to-Real Bridge for Controllable Dexterous World Models

A narrated tour of the method

Decoupling dynamics from rendering

Action-conditioned world models let robots imagine the future consequences of their actions without physical interaction, making them a powerful tool for policy evaluation, planning, and data augmentation. We present Mask2Real-WM, a two-stage action-conditioned world model for dexterous manipulation that decouples pixel prediction into a dynamics model and a rendering model.

The dynamics model predicts future segmentation masks from past masks and a 23-dimensional action sequence (a 6-D Cartesian end-effector pose and 17 hand-joint positions); the rendering model paints photorealistic RGB onto the predicted masks via a ControlNet-augmented SVD backbone. Because segmentation space has a small sim-to-real gap, the dynamics model benefits from large-scale pretraining on over 50 h of synthetic simulation data followed by fine-tuning on approximately 2.5 h of real demonstrations (an avenue largely unexplored for image-space world models).

Experiments on a dexterous pick-and-place benchmark show that both mask conditioning and simulation pretraining are necessary to achieve high mean command-response controllability across the 23 action dimensions, while monolithic baselines capture only coarse end-effector motion.

Synthetic pretraining

>50 hIsaacLab simulation

Real demonstrations

≈2.5 hFine-tuning budget

ID controllability

0.95vs. 0.60 baseline

Decouple what moves where from what it looks like

Predicting dexterous manipulation in raw pixels forces a single model to learn both physics and appearance at once, and the sim-to-real appearance gap blocks the use of cheap synthetic data. Mask2Real-WM instead factorizes the problem into one shared system of two stages, chained autoregressively at inference:

A two-stage world model, told as a picture

Per-dimension controllability, in 3D

Click any joint on the 3D ORCA hand (or a button) to see the world model's prediction when that single action dimension is perturbed. Switch models and samples to compare side by side.

What mask conditioning and sim pretraining buy

The real training data is concentrated around grasping and free-play manipulation and contains only limited isolated single-finger motion. Simulation pretraining alone lifts controllability from ≈0.68→0.85 (ID) and ≈0.51→0.73 (OOD); real fine-tuning then closes the remaining gap to 0.95 ID / 0.87 OOD, with the largest benefit on near-grasp OOD configurations.

Autoregressive rollout, ours vs. monolithic

Autoregressive rollout on a pick-and-place sequence with the cup, comparing Mask2Real-WM against the monolithic baseline.

Behavior unfolding inside the world model

These examples visualize how long-horizon behavior unfolds as ACT, Diffusion Policy, and Flow Matching are rolled out inside Mask2Real-WM. Each video shows both the third-person and wrist-camera views: the upper half is the starting image for each view, and the lower half is the policy rollout.

Cup, banana, cylinder

Qualitative predictions on different manipulation targets.

Cite this work

@article{feingold2026mask2realwm,
  title   = {Mask2Real-WM: Segmentation Masks as a Sim-to-Real Bridge
             for Controllable Dexterous World Models},
  author  = {Feingold, Riccardo Orion and Liconti, Davide and
             Yang, Chenyu and Katzschmann, Robert K.},
  journal = {arXiv preprint arXiv:2607.04546},
  year    = {2026}
}