Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flows

Abstract

Real-world fine-tuning of dexterous manipulation policies remains challenging due to limited real-world interaction budgets and highly multimodal action distributions. Diffusion-based policies, while expressive, do not permit conservative likelihood-based updates during fine-tuning because action probabilities are intractable. In contrast, conventional Gaussian policies collapse under multimodality, particularly when actions are executed in chunks, and standard per-step critics fail to align with chunked execution, leading to poor credit assignment.

We present SOFT-FLOW, a sample-efficient off-policy fine-tuning framework with normalizing flows to address these challenges. The normalizing-flow policy yields exact likelihoods for multimodal action chunks, allowing conservative, stable policy updates through likelihood regularization and thereby improving sample efficiency. An action-chunked critic evaluates entire action sequences, aligning value estimation with the policy’s temporal structure and improving long-horizon credit assignment. We evaluate SOFT-FLOW on two challenging real-world dexterous manipulation tasks: scissors retrieval & tape cutting, and in-hand palm-done cube rotation. On these tasks, SOFT-FLOW achieves stable, sample-efficient adaptation.

Overview

Key idea: keep the policy expressive (multimodal action chunks) while making it tractable (exact log-likelihoods), and align value learning with the same temporal abstraction.

• Normalizing-flow actor models action chunks and provides exact likelihoods for conservative updates.
• Action-chunked critic evaluates entire sequences to improve long-horizon credit assignment.
• Practical pipeline for limited on-robot data: imitation / distillation → critic warm-up → offline RL → online RL.

Method at a Glance

Videos

Videos play at 2x speed by default for convenience. ✔︎ marks successful sub-task completion, ✖︎ marks failure for scissors task.

Real-World: Scissors Retrieval & Tape Cutting

Real-World: In-hand Palm-down Cube Rotation

Results

Scissors Retrieval & Tape Cutting

Starting from imitation learning, SOFT-FLOW improves grasping with offline RL and enables successful cutting after limited online fine-tuning. In the paper’s summary table, SOFT-FLOW reaches 70% grasp and 70% cutting success in the final setting.

In-hand Palm-down Cube Rotation

Online fine-tuning refines a distilled sim policy into robust real-world rotation. The reported peak performance reaches 6.25 rotations/min after about 105 minutes of real-world data.

Citation

@article{yang2026softflow,
  title        = {Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flows},
  author       = {Yang, Chenyu and Tarasov, Denis and Liconti, Davide and Zheng, Hehui and Katzschmann, Robert K.},
  journal      = {arXiv preprint arXiv:2602.09580},
  year         = {2026}
}