GAP: Generalized Adaptation Prior for Dexterous Piano Playing

Jinyi Chen, Hongyi Li, Lei Jiang, Xianwei Liu, Yanhong Liang, Yulin Chen, Yucheng Tao, Yongbin Jin, Hongtao Wang

November 2026

TL;DR Instead of transferring a piano-playing policy from simulation, GAP transfers a network that predicts how a correction to an executed trajectory changes keystroke accuracy. On a tendon-driven hand it raises F1 by 3.6%-23% on three pieces after about 7 minutes each.

What they do

Piano trajectories trained in simulation lose accuracy on a real dexterous hand, and fixing them with residual RL costs a lot of hardware time. GAP learns how to adapt instead of how to act: a Gradient Guidance Network (GGN) predicts how a small correction to an already-executed trajectory would change keystroke accuracy. That prior is pretrained in simulation and reused on hardware, where corrections are updated between repeated executions of a piece.

How it works

Corrections. The initial trajectory, exported from a per-piece RL policy, is sent at 50 Hz. Corrections live on 10 Hz nodes and are interpolated onto it.
GGN. A 2-layer Transformer takes a window of observations around a node (commands, joint positions and velocities, pressed and target keys, fingering) plus nearby corrections, and outputs the local reward change. It is trained by supervised learning on paired executions of the initial and adjusted trajectories.
Adaptation. With observations fixed, 20 Adam steps move the corrections up the GGN gradient; the minimum of two networks limits overestimation.
Two stages. Pretraining uses 4 s segments of three pieces in parallel simulation with randomized zero offsets, damping and delays. On hardware the piece is adapted segment by segment and GGN is fine-tuned on the new pairs.

Results

Simulation, 20 min budget: mean F1 rises from 0.761 to 0.873 with GAP and to 0.807 with HandelBot, about 2.4x the improvement.
Without pretraining, reaching GAP’s few-shot F1 takes 23.2 min instead of 5.3 min. With the target piece held out of pretraining, few-shot F1 is 0.843 versus 0.855.
Real hand (18 joints, on a UR5, Yamaha digital piano): after one iteration of about 7 min, F1 improves by about 16% on Canon and 23% on Für Elise, both started with a deliberate arm offset of about 11 mm and 9 mm, and by 3.6% on Beethoven’s Symphony No. 5.
With continued fine-tuning Beethoven goes from 0.751 to 0.871 in about 50 min; the other two settle near 0.98.

Limitations

GAP refines a given trajectory, so it needs one that already plays most target keys, and it corrects between executions, not during playing. The evaluation covers right-hand excerpts of about 30 s from three pieces and scores keystroke accuracy only, without dynamics or timbre.

Figures

From Figure 1: GAP pretrains a generalized adaptation prior in simulation on multiple pieces under different perturba... — From Figure 1: GAP pretrains a generalized adaptation prior in simulation on multiple pieces under different perturbations, including joint zero offsets, damping, actuator delays, and sensor noise (top), and uses it for fast adaptation on the physical hand (bottom). Green, red, and orange keys denote correct, incorrect, and missed keystrokes.

From Figure 2: Overview of GAP. Both stages share one adaptation process: gradients of the reward change modeled by t... — From Figure 2: Overview of GAP. Both stages share one adaptation process: gradients of the reward change modeled by the Gradient Guidance Network (GGN) iteratively update the action corrections, and paired executions of the initial and adjusted trajectories supervise GGN. The execution and collection module is instantiated in parallel simulation in Stage I and on the physical hand in Stage II.

From Figure 5: F1 scores at each whole-piece evaluation during real-world adaptation of three representative pieces. ... — From Figure 5: F1 scores at each whole-piece evaluation during real-world adaptation of three representative pieces. Evaluations 0 and 1 denote the initial and few-shot trajectories, respectively. Time includes execution, training, and scheduling. Dashed lines show direct-deployment F1 without added offsets or adaptation.

From Figure 7: Real-world adaptation by GAP at 5 instants in Beethoven’s Symphony No. 5. Each column compares the ini... — From Figure 7: Real-world adaptation by GAP at 5 instants in Beethoven’s Symphony No. 5. Each column compares the initial (top) and adapted (bottom) keystrokes at one instant. Green, red, and orange denote correct, incorrect, and missed keystrokes; examples include off-center, overly light, and mistimed strikes.

Abstract

Dexterous hands enable versatile manipulation, but policies trained in simulation often degrade on hardware owing to non-idealities, and the high-dimensional action space makes real-world adaptation costly. Piano playing, which requires precise and coordinated multifinger motion, offers a representative testbed. We propose GAP (Generalized Adaptation Prior for Dexterous Piano Playing), a sample-efficient adaptation framework that learns how to adapt, rather than how to act, during simulation pretraining and transfers this adaptation prior to hardware. Between repeated executions, GAP uses feedback to adjust the actions of the next one: a gradient guidance network learns from paired executions how adjusting actions affects playing performance, and its gradients guide these adjustments. In simulation, GAP yields approximately 2.4 times the F1 scores improvement of the existing fast-adaptation baseline for dexterous piano playing under the same interaction budget. The learned prior also generalizes to musical pieces held out from pretraining. On a physical dexterous hand, the same simulation-pretrained prior improves F1 scores by 3.6%-23% across three pieces of varying difficulty after approximately 7 min of adaptation each. These results suggest a practical route to reduce the real-world interaction needed to deploy dexterous hands.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong