SERNF: Sample-Efficient Real-World Dexterous Fine-Tuning with Normalizing-Flow Policies

Submitted to IEEE Robotics and Automation Letters (RA-L), 2026 · arXiv:2602.09580

Chenyu Yang*, Denis Tarasov*, Davide Liconti, Romain Guntz, Hehui Zheng, Robert K. Katzschmann

*Equal contribution; Soft Robotics Lab, D-MAVT, ETH Zurich

Soft Robotics Lab logo ETH Zurich logo
arXiv PDF Videos BibTeX Code (coming)

SERNF is a sample-efficient, off-policy framework for fine-tuning normalizing-flow (NF) policies directly on real robots. The NF actor models multimodal action chunks, gives exact likelihoods for conservative updates and runs in a single pass; an action-chunked critic aligns value learning with chunked execution. The same recipe improves scissors cutting, palm-down cube rotation and a 48-DoF bimanual packing task with a limited real-world interaction budget.

Video

Abstract

Real-world fine-tuning of dexterous manipulation policies remains challenging due to limited real-world interaction budgets and highly multimodal action distributions. Although diffusion and flow-matching policies model such distributions well, they do not provide tractable exact action likelihoods for conservative policy updates. Moreover, differentiating through their iterative sampling procedures can make end-to-end critic-guided fine-tuning costly and unstable on real robots. Conventional Gaussian policies, by contrast, can collapse under multimodality, particularly when actions are executed in chunks; standard per-step critics are also misaligned with chunked execution.

We present SERNF, a sample-efficient off-policy framework for fine-tuning normalizing-flow (NF) policies. The NF policy represents multimodal action chunks and provides exact likelihoods for conservative regularization. An action-chunked critic evaluates entire action sequences, aligning value estimation with the policy’s temporal structure. We evaluate SERNF on three challenging real-world tasks spanning precise, dynamic, long-horizon, and high-dimensional dexterous manipulation. SERNF increases scissors-cutting success from 0.20 to 0.93 with 60 additional rollouts, reaches 6.25 RPM in cube rotation after 105 minutes of real-world interaction, and autonomously completes a 48-DoF bimanual packing task in 8 of 16 evaluation rollouts after 200 human-in-the-loop rollouts.

Overview

SERNF overview
SERNF couples an exact-likelihood normalizing-flow actor with an action-chunked critic. The policy can be initialized from real-world teleoperation or from simulation and then fine-tuned with limited real-world interaction.

Method at a Glance

I. Policy initialization

Train the NF policy by maximum likelihood on successful demonstrations, or distill it from a simulation teacher.

II. Critic warm-up

Fit the chunked critic with the actor frozen. Best-of-N selection already improves behavior before any policy update.

III. Offline RL

Maximize Q regularized by the exact NF log-likelihood, reusing all data including failed trajectories.

IV. Online RL

Fine-tune on the robot with mixed offline/online batches and a small rollout budget.

SERNF actor-critic architecture
Actor: a conditional NF maps between action chunks and a base Gaussian through invertible attention-based coupling blocks. Critic: a transformer predicts an HL-Gaussian distribution over chunk values; the policy uses the minimum over the critic ensemble.
Real-time chunking inside the RL transition
Real-time chunking (RTC) inside the RL transition: the policy conditions on an observation history and the queued prefix actions of the previous chunk, then predicts H actions.
Real-world setups
Real-world setups. Left: scissors retrieval and tape cutting (Franka + ORCA hand, 24-DoF actions). Middle: palm-down cube rotation (ORCA hand, 17-DoF). Right: bimanual tennis-ball packing (two Franka–ORCA systems, 48-DoF).

Videos

marks successful rollout completion and failure.

Final Results (After Online RL)

Scissors retrieval & tape cutting — after online RL.

In-hand palm-down cube rotation — after online RL.

Bimanual tennis-ball packing — fully autonomous, after online RL.

Real-World: Scissors Retrieval & Tape Cutting

Step 1: Imitation learning (71 successful demos)

Step 2: Offline RL (all 121 trajectories, incl. failures)

Step 3: Online RL (+60 autonomous rollouts)

Show more scissors rollouts

Real-World: In-hand Palm-down Cube Rotation

Step 1: Simulation teacher (PPO + domain randomization)

Step 2: Distilled NF policy (real)

Step 3: After critic warm-up (on online data)

Step 4: After online RL (105 min of real-world data)

Real-World: Bimanual Tennis-Ball Packing (48-DoF)

Two Franka arms with two ORCA hands must pick up a bottle and a tennis ball, insert the ball into the bottle in mid-air, and place the loaded bottle back down. Online rollouts are collected human-in-the-loop (DAgger-style): the operator takes over only after the policy fails.

Online RL data collection: policy rollout → human takeover after failure (2×).

Final policy, fully autonomous (after 200 HIL rollouts): bottle → ball → insert → place.

Robustness to disturbances (never seen in training)

Ball knocked away — the policy re-grasps and completes the pack (1×).

Ball knocked out of the hand — recovery and re-grasp (1×).

Additional Task: Duck Pick-and-Place

From an earlier version of this work (arXiv v1). This task uses only 30 teleoperated demonstrations. The robot must pick the duck from varying table poses and place it into the bowl, making low-data adaptation substantially more challenging.

Step 1: Imitation learning

Step 2: Offline RL

Step 3: Online RL

Show more duck rollouts

Results

Scissors Retrieval & Tape Cutting

Imitation learning grasps about two thirds of the time but rarely cuts. Offline RL lifts grasping from 0.67 to 0.80 by also learning from failed demonstrations. Online RL then teaches the policy to reorient the scissors and cut, taking both grasping and cutting to 0.93 — a more than fourfold increase in cutting from just 60 additional rollouts. The diffusion-RL baseline DPPO, run with the same protocol, fluctuates around its IL performance.

Scissors task success rates for SERNF and DPPO

In-hand Palm-down Cube Rotation

The task cannot be teleoperated, so the NF policy is distilled from a simulation teacher. The distilled policy rarely completes a full turn. After 3k critic-only updates and a short exploration dip, online RL climbs to 6.25 rotations/min after 7k gradient steps and 105 minutes of real-world data, closing a sim-to-real gap that distillation alone does not.

Cube rotation learning curve

Bimanual Tennis-Ball Packing

With a 48-DoF action space and four sequential sub-goals, the IL policy picks the bottle in 6/16 rollouts and never gets past the pickups. Online RL with 200 human-in-the-loop rollouts lifts bottle and ball pickup to 0.81 and 0.75. Packing, which never occurs before online RL, first appears at +8k steps and reaches 8/16 fully autonomous successes at the final checkpoint. The final policy also recovers from distractors and from the ball being knocked out of its hand.

Bimanual packing sub-goal success rates

Additional: Duck Pick-and-Place

With only 30 teleoperated demonstrations, offline RL reaches 68.75% success and online RL with just 15 additional rollouts improves this to 87.5% (75% after imitation learning). Average completion time on commonly solved rollouts drops from 8.86 s (IL) to 7.08 s (offline) and 6.91 s (online).

Duck task learning curve

Baseline Comparison (Scissors Task)

Method Grasping Cutting
ACT (IL)0.530.20
Diffusion Policy (IL)0.670.40
NF (IL)0.670.20
NF IL + online RL data0.130.00
ACT + offline RL0.000.00
DPPO0.400.27
SERNF (offline only)0.800.40
SERNF (full)0.930.93

All IL policies share the dataset, augmentation and frozen DINOv2 encoder. Under imitation alone the NF head is not the strongest; SERNF’s gains come from the RL machinery built on top of it. Simply adding the successful online rollouts to the IL dataset hurts performance.

Citation

@article{yang2026sernf,
  title={SERNF: Sample-Efficient Real-World Dexterous Fine-Tuning with Normalizing-Flow Policies},
  author={Yang, Chenyu and Tarasov, Denis and Liconti, Davide and Guntz, Romain and Zheng, Hehui and Katzschmann, Robert K},
  journal={arXiv preprint arXiv:2602.09580},
  year={2026}
}