I. Policy initialization
Train the NF policy by maximum likelihood on successful demonstrations, or distill it from a simulation teacher.
Submitted to IEEE Robotics and Automation Letters (RA-L), 2026 · arXiv:2602.09580
*Equal contribution; Soft Robotics Lab, D-MAVT, ETH Zurich
SERNF is a sample-efficient, off-policy framework for fine-tuning normalizing-flow (NF) policies directly on real robots. The NF actor models multimodal action chunks, gives exact likelihoods for conservative updates and runs in a single pass; an action-chunked critic aligns value learning with chunked execution. The same recipe improves scissors cutting, palm-down cube rotation and a 48-DoF bimanual packing task with a limited real-world interaction budget.
Real-world fine-tuning of dexterous manipulation policies remains challenging due to limited real-world
interaction budgets and highly multimodal action distributions. Although diffusion and flow-matching
policies model such distributions well, they do not provide tractable exact action likelihoods for
conservative policy updates. Moreover, differentiating through their iterative sampling procedures can make
end-to-end critic-guided fine-tuning costly and unstable on real robots. Conventional Gaussian policies, by
contrast, can collapse under multimodality, particularly when actions are executed in chunks; standard
per-step critics are also misaligned with chunked execution.
We present SERNF, a sample-efficient off-policy framework for fine-tuning normalizing-flow (NF)
policies. The NF policy represents multimodal action chunks and provides exact likelihoods for conservative
regularization. An action-chunked critic evaluates entire action sequences, aligning value estimation
with the policy’s temporal structure. We evaluate SERNF on three challenging real-world tasks spanning
precise, dynamic, long-horizon, and high-dimensional dexterous manipulation. SERNF increases scissors-cutting
success from 0.20 to 0.93 with 60 additional rollouts, reaches 6.25 RPM in cube rotation after
105 minutes of real-world interaction, and autonomously completes a 48-DoF bimanual packing task in
8 of 16 evaluation rollouts after 200 human-in-the-loop rollouts.
Train the NF policy by maximum likelihood on successful demonstrations, or distill it from a simulation teacher.
Fit the chunked critic with the actor frozen. Best-of-N selection already improves behavior before any policy update.
Maximize Q regularized by the exact NF log-likelihood, reusing all data including failed trajectories.
Fine-tune on the robot with mixed offline/online batches and a small rollout budget.
marks successful rollout completion and failure.
Scissors retrieval & tape cutting — after online RL.
In-hand palm-down cube rotation — after online RL.
Bimanual tennis-ball packing — fully autonomous, after online RL.
Step 1: Imitation learning (71 successful demos)
Step 2: Offline RL (all 121 trajectories, incl. failures)
Step 3: Online RL (+60 autonomous rollouts)
Step 1: Simulation teacher (PPO + domain randomization)
Step 2: Distilled NF policy (real)
Step 3: After critic warm-up (on online data)
Step 4: After online RL (105 min of real-world data)
Two Franka arms with two ORCA hands must pick up a bottle and a tennis ball, insert the ball into the bottle in mid-air, and place the loaded bottle back down. Online rollouts are collected human-in-the-loop (DAgger-style): the operator takes over only after the policy fails.
Online RL data collection: policy rollout → human takeover after failure (2×).
Final policy, fully autonomous (after 200 HIL rollouts): bottle → ball → insert → place.
Ball knocked away — the policy re-grasps and completes the pack (1×).
Ball knocked out of the hand — recovery and re-grasp (1×).
From an earlier version of this work (arXiv v1). This task uses only 30 teleoperated demonstrations. The robot must pick the duck from varying table poses and place it into the bowl, making low-data adaptation substantially more challenging.
Step 1: Imitation learning
Step 2: Offline RL
Step 3: Online RL
Imitation learning grasps about two thirds of the time but rarely cuts. Offline RL lifts grasping from 0.67 to 0.80 by also learning from failed demonstrations. Online RL then teaches the policy to reorient the scissors and cut, taking both grasping and cutting to 0.93 — a more than fourfold increase in cutting from just 60 additional rollouts. The diffusion-RL baseline DPPO, run with the same protocol, fluctuates around its IL performance.
The task cannot be teleoperated, so the NF policy is distilled from a simulation teacher. The distilled policy rarely completes a full turn. After 3k critic-only updates and a short exploration dip, online RL climbs to 6.25 rotations/min after 7k gradient steps and 105 minutes of real-world data, closing a sim-to-real gap that distillation alone does not.
With a 48-DoF action space and four sequential sub-goals, the IL policy picks the bottle in 6/16 rollouts and never gets past the pickups. Online RL with 200 human-in-the-loop rollouts lifts bottle and ball pickup to 0.81 and 0.75. Packing, which never occurs before online RL, first appears at +8k steps and reaches 8/16 fully autonomous successes at the final checkpoint. The final policy also recovers from distractors and from the ball being knocked out of its hand.
With only 30 teleoperated demonstrations, offline RL reaches 68.75% success and online RL with just 15 additional rollouts improves this to 87.5% (75% after imitation learning). Average completion time on commonly solved rollouts drops from 8.86 s (IL) to 7.08 s (offline) and 6.91 s (online).
| Method | Grasping | Cutting |
|---|---|---|
| ACT (IL) | 0.53 | 0.20 |
| Diffusion Policy (IL) | 0.67 | 0.40 |
| NF (IL) | 0.67 | 0.20 |
| NF IL + online RL data | 0.13 | 0.00 |
| ACT + offline RL | 0.00 | 0.00 |
| DPPO | 0.40 | 0.27 |
| SERNF (offline only) | 0.80 | 0.40 |
| SERNF (full) | 0.93 | 0.93 |
All IL policies share the dataset, augmentation and frozen DINOv2 encoder. Under imitation alone the NF head is not the strongest; SERNF’s gains come from the RL machinery built on top of it. Simply adding the successful online rollouts to the IL dataset hurts performance.
@article{yang2026sernf,
title={SERNF: Sample-Efficient Real-World Dexterous Fine-Tuning with Normalizing-Flow Policies},
author={Yang, Chenyu and Tarasov, Denis and Liconti, Davide and Guntz, Romain and Zheng, Hehui and Katzschmann, Robert K},
journal={arXiv preprint arXiv:2602.09580},
year={2026}
}