SERNF: Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flows

Abstract

Real-world fine-tuning of dexterous manipulation policies remains challenging due to limited real-world interaction budgets and highly multimodal action distributions. Diffusion-based policies, while expressive, do not permit conservative likelihood-based updates during fine-tuning because action probabilities are intractable. In contrast, conventional Gaussian policies collapse under multimodality, particularly when actions are executed in chunks, and standard per-step critics fail to align with chunked execution, leading to poor credit assignment.

We present SERNF, a Sample-Efficient Reinforcement learning with Normalizing Flows to address these challenges. The normalizing-flow policy yields exact likelihoods for multimodal action chunks, allowing conservative, stable policy updates through likelihood regularization and thereby improving sample efficiency. An action-chunked critic evaluates entire action sequences, aligning value estimation with the policy’s temporal structure and improving long-horizon credit assignment. We evaluate SERNF on two challenging real-world dexterous manipulation tasks: scissors retrieval & tape cutting, and in-hand palm-done cube rotation. On these tasks, SERNF achieves stable, sample-efficient adaptation.

Overview

Key idea: keep the policy expressive (multimodal action chunks) while making it tractable (exact log-likelihoods), and align value learning with the same temporal abstraction.

• Normalizing-flow actor models action chunks and provides exact likelihoods for conservative updates.
• Action-chunked critic evaluates entire sequences to improve long-horizon credit assignment.
• Practical pipeline for limited on-robot data: imitation / distillation → critic warm-up → offline RL → online RL.

Method at a Glance

Videos

marks successful rollout completion and failure.

Final Results (After Online RL)

Real-World: Duck Pick-and-Place

This task uses only 30 teleoperated demonstrations. The robot must pick the duck from varying table poses and place it into the bowl, making low-data adaptation substantially more challenging.

Real-World: Scissors Retrieval & Tape Cutting

Real-World: In-hand Palm-down Cube Rotation

Results

Duck Pick-and-Place

With only 30 teleoperated demonstrations, the duck task is a challenging low-data setting. Even under varying pickup locations and orientations, offline RL reaches 68.75% success, and subsequent online RL with just 15 additional rollouts improves this to 87.5%, compared with 75% after imitation learning. On the subset of rollouts where all three policies succeed, RL also speeds up execution: average completion time drops from 8.86s for imitation learning to 7.08s after offline RL and 6.91s after online RL.

Scissors Retrieval & Tape Cutting

Starting from imitation learning, SERNF improves grasping with offline RL and enables successful cutting after limited online fine-tuning. In the paper’s summary table, SERNF reaches 70% grasp and 70% cutting success in the final setting.

In-hand Palm-down Cube Rotation

Online fine-tuning refines a distilled sim policy into robust real-world rotation. The reported peak performance reaches 6.25 rotations/min after about 105 minutes of real-world data.

Citation

@article{yang2026sernf,
  title={Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flows},
  author={Yang, Chenyu and Tarasov, Denis and Liconti, Davide and Zheng, Hehui and Katzschmann, Robert K},
  journal={arXiv e-prints},
  pages={arXiv--2602},
  year={2026}
}
}