Q-OIL: Optimistic Exploration in Learning from Human Interventions

Pranav Teegavarapu, Sriyash Poddar, Rico Qi, Cleah Winston, Krisha Khandelwal, Aykut Onyol, Natasha Jaques, Abhishek Gupta

November 2026

Coffee pouring on the bimanual YAM, BC+RL vs Q-OIL training, sped up.

TL;DR A study of how real people intervene leads to a progress-based simulated human and to Q-OIL, an RL method that gives human corrections an optimism bonus through a separate critic. On three real Franka tasks it learns with 10-50% higher success and almost 30% less human effort than baselines.

What they do

Methods that learn online from human takeovers are usually developed against simulated humans who intervene as soon as an action is suboptimal and then correct optimally. The authors log real interventions (8 participants supervising a simulated robot on 3 tasks, about 200k transitions) and find otherwise: people step in when progress stalls over a horizon, and their corrections are suboptimal and often follow a different strategy than the final policy. They turn this into a better simulated human and a new learning method, Q-OIL.

How it works

Progress-aware model (PAM). The chance of intervening rises when the value gained over the last k steps falls below a threshold, with both fit to held-out human data. Corrections come from an RL checkpoint with about 50-70% success.
Optimism bonus. Intervention transitions get extra reward, which steers exploration toward human-corrected regions instead of imitating them as targets. A small BC term on intervention actions is added.
Decoupled critics. A task critic learns from the task reward only. An optimistic critic adds the bonus but bootstraps from the task critic, so the bonus never leaks back to the pre-intervention states where the robot was failing.
Built on RLPD for a Franka with RGB inputs, and as a residual policy on a frozen pretrained VLA for a bimanual YAM.

Results

PAM reproduces the real-human ranking of six algorithms (Spearman ρ = 1.0) and predicts held-out interventions with 0.175 precision and 0.606 recall, against 0.114 and 0.449 for an action-suboptimality model.
With four real participants supervising simulated Robomimic tasks (48 runs), Q-OIL beats all baselines and needs fewer interventions.
Real Franka: 10-50% higher success, almost 30% less human effort and on average 90% higher throughput (successes per 1k steps).
YAM with a VLA: full-task success of 50% vs 0% for BC+RL on bag assembly and 70% vs 30% on coffee pouring, with fewer interventions.

Limitations

The human study has 8 users on simulated tasks, and each real-robot run has a single supervisor. All tasks are tabletop, and the Franka policies control only end-effector position and the gripper. The bonus guides only local exploration around corrections.

Videos

Unbutton Shirt on the Franka, BC+RL vs Q-OIL training at 16x.

Bag assembly on the YAM, with the supervisor at the leader arms.

Figures

From Figure 1: Real-user data shows that humans intervene when robot progress stalls and give suboptimal corrections.... — From Figure 1: Real-user data shows that humans intervene when robot progress stalls and give suboptimal corrections. Q-OIL treats interventions as optimistic exploration bonuses while keeping the bonuses from propagating to non-intervention states. Right: the real-world tasks.

From Figure 2: (a) Progress over a horizon separates intervention states better than instantaneous action values. (b)... — From Figure 2: (a) Progress over a horizon separates intervention states better than instantaneous action values. (b) Humans intervene when the value plateaus or declines.

From Figure 5: Overview of Q-OIL. The task critic learns from the task reward; the optimistic critic adds the interve... — From Figure 5: Overview of Q-OIL. The task critic learns from the task reward; the optimistic critic adds the intervention bonus, bootstraps from the task critic and drives policy improvement.

From Figure 8: On three real Franka tasks (Pen-in-Bowl, Peg-Insertion, Unbutton Shirt), Q-OIL achieves higher success... — From Figure 8: On three real Franka tasks (Pen-in-Bowl, Peg-Insertion, Unbutton Shirt), Q-OIL achieves higher success and throughput for a fixed interaction budget, with fewer intervention steps.

Abstract

As robots are deployed in the real world, they will inevitably make mistakes. Human interventions provide valuable feedback for improving their behavior, enabling them to adapt online to new scenarios. However, studying human-in-the-loop robot learning is difficult: real-world experiments are slow and expensive, so most algorithm development relies on simulated human models. We show that common human modeling assumptions do not match how real people intervene in practice. Through analyzing intervention data from human participants, we find that humans tend to intervene when agent progress stalls, and that their corrective actions are often suboptimal. Motivated by these findings, we introduce a progress-based model of human intervention that achieves higher rank correlation with real-user algorithm rankings, enabling realistic algorithm benchmarking and development in simulation. Building on these insights, we propose Q-OIL, a method that adds optimistic bonuses to intervention transitions in reinforcement learning and softly guides exploration toward human corrections. We find that decoupled critics are critical for stability because they prevent the bonuses from backing up to pre-intervention states. As a result, Q-OIL learns more efficiently than competitive baselines, achieving 10-50% higher success in simulation and on real robots, and improving pretrained VLAs on two bimanual manipulation tasks. As we ground both evaluation and algorithms in realistic human behavior, this work provides a practical path toward scalable robot learning from interventions.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong