Mask2Real-WM: Segmentation Masks as a Sim-to-Real Bridge for Controllable Dexterous World Models

Overview video

A narrated tour of the method

Narrated overview of the method, hardware setup, and results.

Abstract

Decoupling dynamics from rendering

Action-conditioned world models let robots imagine the future consequences of their actions without physical interaction, making them a powerful tool for policy evaluation, planning, and data augmentation. We present Mask2Real-WM, a two-stage action-conditioned world model for dexterous manipulation that decouples pixel prediction into a dynamics model and a rendering model.

The dynamics model predicts future segmentation masks from past masks and the 23-DoF action sequence; the rendering model paints photorealistic RGB onto the predicted masks via a ControlNet-augmented SVD backbone. Because segmentation space has a small sim-to-real gap, the dynamics model benefits from large-scale pretraining on over 50 h of synthetic simulation data followed by fine-tuning on fewer than 2.5 h of real demonstrations — an avenue largely unexplored for image-space world models.

Experiments on a dexterous pick-and-place benchmark show that both mask conditioning and simulation pretraining are necessary for per-DoF action controllability across all 23 degrees of freedom, while monolithic baselines collapse to coarse end-effector motion.

Control space

6DoFarm + 17DoFhand = 23DoFcontrolled

A 6-DoF end-effector pose and 17 ORCA hand joints are driven as one action sequence — every degree of freedom conditions the dynamics model directly, rather than through a coarse end-effector proxy.

Synthetic pretraining: >50 hIsaacLab simulation
Real demonstrations: <2.5 hFine-tuning budget
ID controllability: 0.95vs. 0.60 baseline

Key idea

Decouple what moves where from what it looks like

Predicting dexterous manipulation in raw pixels forces a single model to learn both physics and appearance at once — and the sim-to-real appearance gap blocks the use of cheap synthetic data. Mask2Real-WM instead factorizes the problem into one shared system of two stages, chained autoregressively at inference:

WM1 · Dynamics

Masks from actions

An action-conditioned video-diffusion model (SVD backbone) predicts future segmentation masks from past masks and the 23-DoF action sequence. Masks have a tiny sim-to-real gap, so WM1 is pretrained on >50 h of IsaacLab data, then fine-tuned on real data with LoRA.

WM2 · Rendering

RGB from masks

A ControlNet-augmented SVD backbone paints photorealistic two-view RGB onto WM1's predicted masks. Trained on real data only (<2.5 h), since appearance is where the sim-to-real gap is largest. A lightweight CNN encodes masks before ControlNet injection.

Segmentation as a bridge

Masks are nearly domain-invariant, enabling synthetic pretraining that is impractical in RGB space.

Per-DoF controllability

Each of the 23 joints responds faithfully and independently — no coupling collapse to coarse end-effector motion.

Automatic labels

SAM 3 produces masks from text prompts alone, supervising the intermediate representation at scale.

Method

A two-stage world model, told as a picture

Hardware setup: Franka Panda arm with ORCA hand, workspace and wrist cameras — **Hardware.** A 7-DoF Franka Emika Panda arm with the 17-DoF ORCA hand (23 DoF total), observed from a fixed workspace camera and a wrist-mounted camera that captures finger–object contacts in a tilted-wall arena.

Action coverage across 23 DoF, simulation versus real — **Action coverage: sim vs. real.** Per-dimension action range across all 23 DoF. Simulation (>50 h) spans a far wider range than real demonstrations (<2.5 h), motivating large-scale synthetic pretraining of WM1.

Segmentation masks in simulation versus reality look nearly identical — **The sim-to-real bridge.** Segmentation masks (hand: green, object: red, background: black) look nearly identical across simulation and reality, unlike raw RGB — this small domain gap is what makes synthetic pretraining of the dynamics model effective.

Interactive demo

Per-DoF controllability, in 3D

Click any joint on the 3D ORCA hand — or a button — to see the world model's prediction when that single action component is perturbed. Switch models and samples to compare side by side.

Compare WM + LoRA (ours), WM Mid-train, WM Real-Only, and the monolithic Baseline across 23 degrees of freedom. The 3D model loads from STL meshes; videos stream on click. Open full screen ↗

Results

What mask conditioning and sim pretraining buy

Isolated single-finger motion is absent from the real training set, which predominantly contains cube-grasping. Simulation pretraining alone lifts controllability from ≈0.68→0.85 (ID) and ≈0.51→0.73 (OOD); real fine-tuning then closes the remaining gap to 0.95 ID / 0.87 OOD, with the largest benefit on near-grasp OOD configurations.

Controllability results across models and degrees of freedom — **Action controllability.** *Left:* model responses to sinusoidal perturbation of individual action components. *Right:* mean controllability score for ID (top) and OOD (bottom). Our full model (WM1 sim→real, WM2 real) reaches ≈0.95 ID and ≈0.87 OOD; the monolithic baseline falls below 0.5 on OOD.

**Perceptual metrics across WM1 training configurations.** PSNR, SSIM, LPIPS on ID (top) and LPIPS on three OOD splits — *No Object*, *Random Play*, *Background* (bottom) — for real-only, sim-only, and sim-then-real (ours) WM1, all paired with the same WM2. Sim pretraining substantially reduces OOD degradation while real fine-tuning maintains strong ID performance.

WM2 conditioning ablation measured in LPIPS — **WM2 conditioning ablation (LPIPS↓).** Mask conditioning via ControlNet is the dominant driver of spatial sharpness; adding action conditioning further tightens motion consistency. Masks alone already support competitive quality and zero-shot deployment decoupled from the action space.

Long-horizon rollouts

Autoregressive rollout, ours vs. monolithic

Autoregressive rollout on a pick-and-place sequence with the cup, comparing Mask2Real-WM against the monolithic baseline.

Mask2Real-WM (ours) — sharp contacts, stable object identity

Monolithic baseline — blur and drift accumulate

Long-horizon mask-to-RGB rollout strip, ours — **Ours** — long-horizon mask→RGB rollout strip.

Long-horizon rollout strip, baseline — **Baseline** — long-horizon rollout strip; note the loss of finger and object detail over time.

Policy rollouts

Behavior unfolding inside the world model

These examples visualize how long-horizon behavior unfolds as ACT, Diffusion Policy, and Flow Matching are rolled out inside Mask2Real-WM. Each video shows both the third-person and wrist-camera views: the upper half is the starting image for each view, and the lower half is the policy rollout.

ACT · Sequence A

Diffusion Policy · Sequence A

Flow Matching · Sequence A

ACT · Sequence B

Diffusion Policy · Sequence B

Flow Matching · Sequence B

Generalization

Cup, banana, cylinder

Qualitative predictions on different manipulation targets.

Qualitative prediction: banana — **Banana**

Qualitative prediction: cylinder — **Cylinder**

BibTeX

Cite this work

@inproceedings{feingold2026mask2realwm,
  title     = {Mask2Real-WM: Segmentation Masks as a Sim-to-Real Bridge
               for Controllable Dexterous World Models},
  author    = {Feingold, Riccardo Orion and Liconti, Davide and
               Yang, Chenyu and Katzschmann, Robert K.},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2026}
}