Supplementary Material
Mask2Real-WM

A Baseline Comparison & the Effect of Simulation Midtraining

What does large-scale simulation midtraining of WM₁ contribute to the full system?

We compare four regimes that all share the same WM₂ renderer: the single-stage baseline (Ctrl-World, fine-tuned end-to-end on real RGB), and three two-stage models whose WM₁ is trained on (i) real data only, (ii) simulation only (midtraining), and (iii) simulation midtraining then real LoRA fine-tuning (sim+real, ours).

Appearance metrics alone are misleading

Sharpness exposes the blur

Mask-space quality reveals the generalization benefit

Real-only training fits the ID distribution best, but simulation midtraining followed by fine-tuning generalizes best under distribution shift (OOD PSNR 21.7 vs. 20.5), because the synthetic corpus covers contact-rich motions absent from the small real dataset. Long-horizon rollout strips comparing ours vs. the baseline are shown on the main page.

B Per-Component Action Controllability

Does the model respond to each action dimension independently?

We drive a single action dimension at a time with a sinusoid (holding the others fixed) and roll the model out for all 23 dimensions: the 6-DoF end-effector pose (dims 0–5: x, y, z, roll, pitch, yaw) and the 17 ORCA hand joints (dims 6–22). The model produces a coherent, dimension-specific response across the entire action space.

C WM₂ Conditioning Ablation

Ablating WM₂'s two conditioning signals: the ControlNet mask branch and the frame-wise action cross-attention.

Standard pixel metrics (PSNR, SSIM, LPIPS) reward predictions that minimise per-pixel error, biasing them towards blurry, temporally averaged outputs. The actions-only variant therefore appears competitive on these metrics while producing visibly degraded video.

Masks only (no action conditioning) is the primary driver of spatial sharpness and additionally enables zero-shot deployment decoupled from the robot's action space. Masks + actions (ours) yields the best ID pixel metrics and competitive OOD performance while keeping the sharpness gains. The LPIPS summary appears on the main page.

D Error Accumulation in Autoregressive Rollouts

In autoregressive rollout each predicted frame conditions the next, so small mistakes compound — and WM₁ mask errors propagate into WM₂.

E Failure Mode Examples

The current bottleneck is WM₁'s object-state prediction: it reliably captures hand dynamics but is less certain about the object.

F Generalization to Objects Unseen in Real Data

The real demos are almost exclusively red-cube manipulation. We evaluate the full pipeline zero-shot on a banana, cup and cylinder — objects WM₁ has seen in simulation but that WM₂ (real-only) has never rendered.

Because WM₁ operates in segmentation space and is largely agnostic to appearance, the dynamics model transfers directly; only WM₂ must render the new appearance.

The baseline scores higher on per-pixel appearance metrics owing to the blurry-prediction bias, but WM₁ maintains high segmentation-space fidelity (SSIM ≈ 0.86) on these objects — the factorization transfers without retraining the dynamics model; only WM₂ would need updating to match new textures. Qualitative rollouts for each object (ours vs. baseline) are on the main page.

G WM₂ Conditioned on Ground-Truth Masks

Is WM₁'s mask prediction quality the primary bottleneck? We feed WM₂ oracle SAM 3 masks instead of WM₁'s predictions.

H CNN Mask Encoder vs. VAE Encoder for ControlNet

Why a dedicated CNN encoder rather than the SVD VAE encoder to condition ControlNet?

We measure a boundary-response score — the ratio of feature-gradient magnitude at segmentation-class boundaries to that in flat interiors (>1 means boundary-selective features). Across 128 frames the CNN attains 4.41 ± 0.74 vs. 2.98 ± 0.43 for the VAE — class boundaries encoded ≈48% more strongly, on every frame.

I Dataset Statistics

A large, contact-focused simulation corpus paired with a small real dataset.

The simulation corpus contains 13,950 episodes totalling 53.7 h at 5 fps across five objects (banana, cup, cylinder, dice, torus). Episodes are short and contact-focused (mean 13.9 s). The real dataset comprises 218 episodes totalling 2.5 h, with bimodal lengths: 198 short manipulation rollouts (0.71 h) and 20 long free-play sequences (1.80 h), predominantly red-cube grasping.

J Implementation Details

Real-world rig. A 7-DoF Franka Emika Panda arm with the 17-DoF ORCA hand (23 actuated DoF) inside a fixed arena, observed by a static third-person camera and a wrist-mounted camera (two Luxonis OAK-D RGB cameras). The setup figure is on the main page.

Simulation assets. Default Franka Panda URDF and the ORCA hand USD; the arena is reconstructed from physical measurements. No material, lighting or texture tuning is applied.

System identification. To reduce the kinematic gap between the simulated and physical ORCA hand, per-joint PD gains and friction coefficients are optimized with CMA-ES (following Bjelonic et al. 2025) to minimise tracking error under a chirp command. After calibration the simulated trajectory closely follows the measured one across the full frequency sweep — sufficient for the joint-angle-driven segmentation representation to stay accurate.

SAM 3 prompt engineering

For the third-person view, the prompts "hand" and the object name (e.g. "red cube") are used. For the wrist view, a short initialization clip is prepended in which the hand moves clearly into frame; the bounding box from this clip seeds SAM 3's tracker for the rest of the sequence.

CNN mask encoder. Three convolutional blocks (Conv2d + GroupNorm + SiLU) with strided downsampling, a transposed convolution to restore resolution, and a final 1×1 convolution to match the ControlNet feature dimension. Operating on decoded pixel-space masks rather than VAE latents is key — see H.

K Hyperparameters

L Future Work & Policy Rollouts in Imagination

Scaling the data generator, conditioning on camera intrinsics/extrinsics, multi-object segmentation vocabularies, and distilling both stages into few-step samplers for near-real-time rollouts.

We roll out three policies trained on the real robot — a flow-matching policy, a diffusion policy, and an ACT policy (all trained on red-cube grasps and placements onto the red ramp) — inside both Mask2Real-WM and the baseline world model. Using Mask2Real-WM as a policy evaluator — scoring candidate policies by their predicted outcomes in imagination — is a natural extension we leave to future work.

Each: 100-frame rollout (5 frames predicted per step), 4 frames shown ≈33 apart; both views, lower half generated.

← Back to the main page