Rapid Learning of Dexterous In-Hand Pen Writing through Real-Time Jacobian Estimation

Kai Stewart^†, Yasunori Toshimitsu^†, Robert K. Katzschmann^*

Preprint published September 2026 | All authors are from the Soft Robotics Lab, D-MAVT, ETH Zurich, Switzerland

† Equal contribution * Corresponding author

In this research, we present an embodied control approach based on real-time task Jacobian estimation of the combined hand and object system on the physical robot. Using only the CPU on a laptop, the proposed controller begins in-hand pen writing after just 18 s of initialization and continues to adapt online, without an analytic hand–object kinematic/contact model, simulation training, or precollected task demonstrations. We demonstrate that the same estimator/controller formulation works on three anthropomorphic robotic hand systems (one physical, two simulated) to show human-like, in-hand articulation of a grasped pen by an embodiment-independent formulation.

This work showcases an alternative to compute- and data-heavy approaches such as RL and IL for achieving dexterous manipulation through computationally simple and data-efficient algorithms.

Try it yourself!

The hand starts off with zero pre-training, kinematic/contact model, nor precollected demonstrations. After a short sequence of movements to learn its grip, it can write anything you tell it to!

Simulation preview of the Shadow Hand holding a pen

Runs in your browser · downloads about 14 MB when opened

Learning the grip

Tracking error —

Drag to orbit · scroll to zoom

Pen-tip trajectory10 × 10 mm · 2 mm grid

Draw single-stroke paths for the robot to follow

TargetActual pen tip

Jacobian estimate

World frame · m/rad

Validated on three different hands

Physical ORCA hand: Uncut video of full writing sequence

At the start of the video, the robot has no structural knowledge of itself or the object it holds. It learns how its finger movements affect the pen through a short initialization sequence, then continues adapting as it writes.

Same control method, applied to 2 more hands in simulation

Shadow Hand (4x video)

Wuji Hand (4x video)

The simulation environment which implements the proposed controller is available on GitHub. The main code (sim.py) contains all code for the Shadow Hand demo and is kept deliberately minimal (< 400 loc), both as a demonstration of the controller's simplicity and to improve readability and extensibility. Performance was better on the Shadow Hand, which includes the wrist as part of its degrees of freedom.

System architecture

A webcam tracks the pen tip by extrapolating its position from an ArUco marker attached to the pen. After an initial excitation phase in which the hand is moved through a series of predefined grips, the controller continually updates the task Jacobian estimate and converts the desired pen motion into finger commands, while stabilizing the grip.

Learning the Jacobian online

The task Jacobian J maps finger-joint velocities to pen-tip velocity:

\dot{\mathbf{x}} = J\dot{\mathbf{q}}

We learn this map directly from motion. The estimator uses the previous commanded joint increment divided by the elapsed time as its joint-velocity input, together with the observed pen-tip velocity. A recursive least-squares update continuously updates the estimate of J using the prediction error:

\begin{gathered}\mathbf{p}\leftarrow\mathbf{p}/\lambda\\J\leftarrow J+\frac{(\dot{\mathbf{x}}-J\dot{\mathbf{q}})(\mathbf{p}\odot\dot{\mathbf{q}})^\top}{\mathbf{p}^\top(\dot{\mathbf{q}}\odot\dot{\mathbf{q}})+r}\end{gathered}

Here p stores the diagonal covariance, λ is the forgetting factor, r is the observation-noise variance, and ⊙ denotes element-wise multiplication.

Following the path while maintaining the grip

The estimated Jacobian is then used in a kinematic pen tip tracking controller. A PID controller with path-velocity feedforward produces the desired pen velocity v_cmd. We convert it to finger motion with a damped pseudoinverse of the Jacobian J⁺, and also use its nullspace N_⊥ to pull the joints toward the initial hand pose to stabilize the grip:

\begin{gathered}J^+=J^\top(JJ^\top+\varepsilon I)^{-1}\\N_\perp=I_N-J^+J\end{gathered}

\Delta\mathbf{q}=\bigl(J^+\mathbf{v}_{\mathrm{cmd}}+k_{\mathrm{pb}}N_\perp(\mathbf{q}_0-\mathbf{q})\bigr)\Delta t

The first term follows the writing trajectory, while the second stabilizes the grip while minimizing interference with the pen motion. Here ε sets the damping, k_pb sets the grip pullback strength, and q₀ is the initial joint pose.

The whole controller, including the vision system, is lightweight enough to run on a laptop CPU.

Cite (BibTeX)

@misc{stewart2026rapidlearningdexterousinhand,
      title={Rapid Learning of Dexterous In-Hand Pen Writing through Real-Time Jacobian Estimation},
      author={Kai Stewart and Yasunori Toshimitsu and Robert K. Katzschmann},
      year={2026},
      eprint={2609.11775},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2609.11775},
}