Supplementary Material for Mask2Real-WM

Baseline comparison & the effect of simulation midtraining

What does large-scale simulation midtraining of WM₁ contribute to the full system?

We compare four regimes that all share the same WM₂ renderer: the single-stage baseline (Ctrl-World, fine-tuned end-to-end on real RGB), and three two-stage models whose WM₁ is trained on (i) real data only, (ii) simulation only (midtraining), and (iii) simulation midtraining then real LoRA fine-tuning (sim+real, ours).

Per-component action controllability

Does the model respond to each action dimension independently?

We drive a single action dimension at a time with a sinusoid (holding the others fixed) and roll the model out for all 23 dimensions: the 6-DoF end-effector pose (dims 0–5: x, y, z, roll, pitch, yaw) and the 17 ORCA hand joints (dims 6–22). The model produces a coherent, dimension-specific response across the entire action space.

WM₂ conditioning ablation

Ablating WM₂'s two conditioning signals: the ControlNet mask branch and the frame-wise action cross-attention.

Standard pixel metrics (PSNR, SSIM, LPIPS) reward predictions that minimise per-pixel error, biasing them towards blurry, temporally averaged outputs. The actions-only variant therefore appears competitive on these metrics while producing visibly degraded video.

Error accumulation in autoregressive rollouts

In autoregressive rollout each predicted frame conditions the next, so small mistakes compound — and WM₁ mask errors propagate into WM₂.

Failure mode examples

The current bottleneck is WM₁'s object-state prediction: it reliably captures hand dynamics but is less certain about the object.

Generalization to objects unseen in real data

The real demos are almost exclusively red-cube manipulation. We evaluate the full pipeline zero-shot on a banana, cup and cylinder — objects WM₁ has seen in simulation but that WM₂ (real-only) has never rendered.

Because WM₁ operates in segmentation space and is largely agnostic to appearance, the dynamics model transfers directly; only WM₂ must render the new appearance.

WM₂ conditioned on ground-truth masks

Is WM₁'s mask prediction quality the primary bottleneck? We feed WM₂ oracle SAM 3 masks instead of WM₁'s predictions.

CNN mask encoder vs. VAE encoder for ControlNet

Why a dedicated CNN encoder rather than the SVD VAE encoder to condition ControlNet?

We measure a boundary-response score — the ratio of feature-gradient magnitude at segmentation-class boundaries to that in flat interiors (>1 means boundary-selective features). Across 128 frames the CNN attains 4.41 ± 0.74 vs. 2.98 ± 0.43 for the VAE — class boundaries encoded ≈48% more strongly, on every frame.

Dataset statistics

A large, contact-focused simulation corpus paired with a small real dataset.

The simulation corpus contains 13,950 episodes totalling 53.7 h at 5 fps across five objects (banana, cup, cylinder, dice, torus). Episodes are short and contact-focused (mean 13.9 s). The real dataset comprises 218 episodes totalling 2.5 h, with bimodal lengths: 198 short manipulation rollouts (0.71 h) and 20 long free-play sequences (1.80 h), predominantly red-cube grasping.

Implementation details

Real-world rig. A 7-DoF Franka Emika Panda arm with the 17-DoF ORCA hand (23 actuated DoF) inside a fixed arena, observed by a static third-person camera and a wrist-mounted camera (two Luxonis OAK-D RGB cameras). The setup figure is on the main page.

Simulation assets. Default Franka Panda URDF and the ORCA hand USD; the arena is reconstructed from physical measurements. No material, lighting or texture tuning is applied.

System identification. To reduce the kinematic gap between the simulated and physical ORCA hand, per-joint PD gains and friction coefficients are optimized with CMA-ES (following Bjelonic et al. 2025) to minimise tracking error under a chirp command. After calibration the simulated trajectory closely follows the measured one across the full frequency sweep — sufficient for the joint-angle-driven segmentation representation to stay accurate.

SAM 3 prompt engineering

For the third-person view, the prompts "hand" and the object name (e.g. "red cube") are used. For the wrist view, a short initialization clip is prepended in which the hand moves clearly into frame; the bounding box from this clip seeds SAM 3's tracker for the rest of the sequence.

Hyperparameters

Future work & policy rollouts in imagination

Scaling the data generator, conditioning on camera intrinsics/extrinsics, multi-object segmentation vocabularies, and distilling both stages into few-step samplers for near-real-time rollouts.

We roll out three policies trained on the real robot — a flow-matching policy, a diffusion policy, and an ACT policy (all trained on red-cube grasps and placements onto the red ramp) — inside both Mask2Real-WM and the baseline world model. Using Mask2Real-WM as a policy evaluator — scoring candidate policies by their predicted outcomes in imagination — is a natural extension we leave to future work.