A two-stage action-conditioned world model that decouples dynamics from rendering, using segmentation masks to unlock large-scale synthetic pretraining for 23-DoF dexterous manipulation.
Action-conditioned world models enable robots to imagine the future consequences of their actions without physical interaction, making them a powerful tool for policy evaluation, planning, and data augmentation. We present Mask2Real-WM, a two-stage action-conditioned world model for dexterous manipulation that decouples pixel prediction into a dynamics model and a rendering model.
The dynamics model predicts future segmentation masks from past masks and the 23-DoF action sequence; the rendering model paints photorealistic RGB onto the predicted masks via a ControlNet-augmented SVD backbone. Because segmentation space has a small sim-to-real gap, the dynamics model benefits from large-scale pretraining on over 50 h of synthetic simulation data followed by fine-tuning on fewer than 2.5 h of real demonstrations — an avenue largely unexplored for image-space world models.
Experiments on a dexterous pick-and-place benchmark show that both mask conditioning and simulation pretraining are necessary for per-DoF action controllability across all 23 degrees of freedom, while monolithic baselines collapse to coarse end-effector motion.
Decouple what moves where from what it looks like.
Predicting dexterous manipulation in raw pixels forces a single model to learn both physics and appearance at once — and the sim-to-real appearance gap blocks the use of cheap synthetic data. Mask2Real-WM instead factorizes the problem into two stages chained autoregressively at inference:
An action-conditioned video-diffusion model (SVD backbone) predicts future segmentation masks from past masks and the 23-DoF action sequence. Masks have a tiny sim-to-real gap, so WM1 is pretrained on >50 h of IsaacLab data, then fine-tuned on real data with LoRA.
A ControlNet-augmented SVD backbone paints photorealistic two-view RGB onto WM1's predicted masks. Trained on real data only (<2.5 h), since appearance is where the sim-to-real gap is largest. A lightweight CNN encodes masks before ControlNet injection.
Masks are nearly domain-invariant, enabling synthetic pretraining that is impractical in RGB space.
Each of the 23 joints responds faithfully and independently — no coupling collapse.
SAM 3 produces masks from text prompts alone, supervising the intermediate representation at scale.
Click any joint on the 3D ORCA hand (or a button) to see the world model's prediction when that single action component is perturbed. Switch models and samples to compare side by side.
Compare WM + LoRA (ours), WM Mid-train, WM Real-Only, and the monolithic Baseline across 23 degrees of freedom. The 3D model loads from STL meshes; videos stream on click.
Isolated single-finger motion is absent from the real training set, which predominantly contains cube-grasping. Simulation pretraining alone lifts controllability from ≈0.68→0.85 (ID) and ≈0.51→0.73 (OOD); real fine-tuning then closes the remaining gap to 0.95 ID / 0.87 OOD, with the largest benefit on near-grasp OOD configurations.
Autoregressive rollout on a pick-and-place sequence (cup). Ours vs. the monolithic baseline.
Real policies rolled out through Mask2Real-WM on out-of-distribution episodes. Three policy families — ACT, Diffusion Policy, and Flow Matching — evaluated through the same world model, with predicted-vs-real PSNR / SSIM. Videos show the wrist-camera view.
Qualitative predictions on different manipulation targets.
@inproceedings{feingold2026mask2realwm,
title = {Mask2Real-WM: Segmentation Masks as a Sim-to-Real Bridge
for Controllable Dexterous World Models},
author = {Feingold, Riccardo Orion and Liconti, Davide and
Yang, Chenyu and Katzschmann, Robert K.},
booktitle = {Conference on Robot Learning (CoRL)},
year = {2026}
}