HRI and teleoperation
Human-robot interaction, shared autonomy and the teleoperation systems used to collect data.
Start here: seminal papers
Cutaneous Force Feedback as a Sensory Subtraction Technique in HapticsSeminal
August 2011
Replace a haptic device's force feedback with fingertip pressure from wearable cable-driven devices ("sensory subtraction") in simulated needle insertion. With 16 users it overshoots the target less than a visual force display, and under a 50 ms delay it stays stable while full haptic feedback oscillates.
Efficient Model Learning for Human-Robot Collaborative TasksSeminal
May 2014
Cluster human-team demonstrations into a few "human types", learn a reward per type with inverse RL, and plan with a MOMDP that treats the type of a new partner as hidden. Clustering matches an expert's labels 96.5% of the time on a place-and-drill task, and the policy holds its reward as human actions deviate from the demonstrations.
Autonomy Infused Teleoperation with Application to BCI ManipulationSeminal
March 2015
A shared-control system recognizes objects, infers which grasp the user is reaching for and blends the user's command with an autonomous one, so two people with intracortical brain-computer interfaces can control a 7-DoF arm. On adapted Action Research Arm Test tasks, the subjects completed none of their 15 trials each under direct control, versus 13/15 and 8/15 with assistance.
Shared Autonomy via Hindsight Optimization★ Must read
March 2015
Shared autonomy as a POMDP over the user's unknown goal, solved approximately with hindsight optimization so the robot assists toward all likely goals at once. In a 10-user grasping study, users were significantly faster (p=0.006) and used less input than with predict-then-blend, yet trended toward preferring the blend baseline.
Cooperative Inverse Reinforcement Learning★ Must read
June 2016
Frames value alignment as a cooperative game in which only the human knows the reward and both players are paid by it, reduces optimal play to a POMDP over the robot's belief, and proves that expert demonstrations are not the best way to teach an IRL robot. In a gridworld, "instructive" demonstrations beat expert ones on regret over 500 sampled reward functions.
Game-Theoretic Modeling of Human Adaptation in Human-Robot CollaborationSeminal
January 2017
The robot models its human teammate as learning the true payoffs of only the robot action just played, and plans when to fail on purpose to reveal its limits. In a 60-person table-clearing study this gave 42-52% more team reward than a robot that assumes the human adapts completely.
Human-Robot Mutual Adaptation in Shared Autonomy★ Must read
January 2017
A joystick-driven assistive arm infers both the user's goal and how willing they are to adapt, then nudges adaptable users toward a better goal and gives in to stubborn ones. In a 51-person study, 75% of users in this condition switched to the better goal at least once, and they trusted the robot more than one that always ignored them.
Enabling Robots to Communicate their ObjectivesSeminal
February 2017
A simulated self-driving car chooses which of its own behaviors to show people so they learn its objective, modeling the viewer as an approximate inverse-RL learner rather than an exact one. With the best learner model plus examples covering every driving strategy, Mechanical Turk users predicted the car's behavior in new scenes significantly better than after random examples (p = 0.049).
Deep reinforcement learning from human preferences★ Must read
June 2017
A human picks the better of two short clips of the agent, a neural reward model is fit to those choices, and RL optimizes that learned reward. With 700 comparisons it nearly matches RL on the true reward across eight MuJoCo tasks, and it learns a Hopper backflip from 900 comparisons in under an hour.
Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality TeleoperationSeminal
October 2017
A consumer VR headset and hand controllers are used to teleoperate a PR2, and behavioral cloning on the recorded demos learns pixel-to-action policies for ten real manipulation tasks. With under 30 minutes of demonstrations per task, success rates range from 80% to 98.9%.
Social Attention: Modeling Attention in Human CrowdsSeminal
October 2017
A pedestrian trajectory predictor that learns soft attention over everyone in the crowd instead of only nearby neighbours, built as a structural RNN over a spatio-temporal graph. On the ETH and UCY datasets it cuts average displacement error from 0.37 m (Social LSTM) to 0.30 m.
Trajectory Deformations from Physical Human-Robot InteractionSeminal
October 2017
Add an online deformation of the robot's future desired trajectory to impedance control, so a human's push changes where the robot plans to go and not just where it is. In a 10-subject study on a wrist exoskeleton, users applied about half the torque (2.25 vs 4.64 Nm) and moved more smoothly than with impedance control alone.
Trust-Aware Decision Making for Human-Robot Collaboration: Model Learning and PlanningSeminal
January 2018
A robot learns a POMDP with human trust as a hidden state from online user data, then orders its actions to build or lower that trust. In a real-robot table-clearing study with 20 participants, human interventions fell to zero on the can and 71% on the wine glass compared with a policy that ignores trust.
Shared Autonomy via Deep Reinforcement LearningSeminal
February 2018
A deep Q-learning copilot reads the user's raw control input as an observation and executes the near-optimal action closest to it, so it can assist without knowing the dynamics, the user's policy or the goal. In a 12-person Lunar Lander study, every participant succeeded more often and crashed less with the copilot (p < 0.0001).
Probabilistically Safe Robot Planning with Confidence-Based Human PredictionsSeminal
May 2018
A robot treats the rationality coefficient of its human-motion model as a measure of confidence in that model, infers it online, and plans around the resulting predictions with a 1% collision-probability limit. On 48 recorded walking trajectories, this kept a simulated quadcopter farther from people or got it to its goal sooner than fixed-confidence predictions when the person avoided an unmodeled obstacle or headed to an unmodeled goal.
Gaze-based, Context-aware Robotic System for Assisted Reaching and GraspingSeminal
September 2018
The user looks at an object and a UR10 arm plus a robotic glove reach for it, grasp it and use it, sequenced by a finite state machine built on action grammars. 3D gaze is estimated to within 4.68 cm, and with 5 users every reach succeeded, pick-and-place succeeded 96% of the time and pick-pour-place 76%.
Crowd-Robot Interaction: Crowd-aware Robot Navigation with Attention-based Deep Reinforcement LearningSeminal
September 2018
A deep RL value network for crowd navigation that encodes every robot-human pair, adds a local map of each human's neighbors, and pools the pairs with self-attention. In simulated crowds that ignore the robot it reaches the goal in all 500 test cases, versus 0.95 for an LSTM baseline and 0.78 for CADRL.
HG-DAgger: Interactive Imitation Learning with Human ExpertsSeminal
October 2018
A DAgger variant where the human expert decides when to take over from the learner and keeps full control until handing back, with labels recorded only during those takeovers. On a real car weaving between obstacles, the HG-DAgger policy had no collisions or road departures, against 1 collision and 6 departures for behavioral cloning.
Expressing Robot IncapabilitySeminal
October 2018
When a robot cannot finish a task, it plays an optimized "attempt" in which its elbow, shoulder and base mimic the motion its stuck end-effector cannot make. In an online study with 120 people, 95% picked the robot's correct goal, against 60% when the robot simply repeated its failing action.
RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through ImitationSeminal
November 2018
A crowdsourcing platform where remote workers teleoperate a simulated robot arm in 6-DoF by moving an iPhone while watching a video stream in their browser. In 22 hours of use it collected over 2200 successful demonstrations, and using 1000 of them raises sparse-reward RL return on can picking from 0 to 641 out of 1000.
Controlling Assistive Robots with Learned Latent Actions★ Must read
September 2019
They train a state-conditioned VAE on demonstrations so that a 1- or 2-DoF joystick input decodes into a full action for a 7-DoF assistive arm. In an eating study users reached the intended morsel in 44 of 50 trials, more often than with shared autonomy, and in a cooking study they finished faster with less joystick input than with end-effector control.
DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm SystemSeminal
October 2019
A glove-free teleoperation system that drives a KUKA arm and Allegro hand (23 actuated joints) from four depth cameras watching the operator's bare hand. Two pilots ran 15 dexterous tasks, such as pulling money out of a wallet, and 141 of 150 trials succeeded.
Assistive Gym: A Physics Simulation Framework for Assistive Robotics★ Must read
October 2019
An open-source PyBullet framework with six simulated caregiving tasks, a human model with impairments and preferences, and PPO baselines for four commercial robots. Training the simulated person as an active partner raised the best robot's task success by 30.4 points on average.
On the Utility of Learning about Humans for Human-AI CoordinationSeminal
October 2019
In a simplified two-player Overcooked kitchen, agents trained only with themselves (self-play, population-based training, joint planning) coordinate much worse with humans than a PPO agent trained against a behavior-cloned human model. In a study with 60 users, the human-aware agent scored significantly higher than self-play (p=.01) and PBT (p<.01).
Shared Autonomy with Learned Latent ActionsSeminal
May 2020
A joystick-to-arm decoder conditioned on the robot's belief over goals is combined with shared autonomy, so the same low-DoF inputs first steer toward a goal and then control fine manipulation there. In a 10-person eating study with a 7-DoF arm, this combination had the lowest total and idle time on both tasks (p<.05).
Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and PreferencesSeminal
June 2020
DemPref initializes a belief over a robot's reward from a few human demonstrations, then refines it with pairwise preference queries chosen by information gain. With one demonstration it reaches in 10 queries the reward alignment that preferences alone need 20-35 queries for, and users rated its queries significantly easier to answer than volume-removal queries.
Object Handovers: a Review for RoboticsSeminal
July 2020
A survey of human-robot object handovers that splits each exchange into a pre-handover phase and the physical handover, and compares 51 papers with real-robot experiments. It proposes a minimal evaluation set: success rate, total handover time and the receiver's task time, plus Likert questions on fluency, trust and working alliance.
Learning Latent Representations to Influence Multi-Agent InteractionSeminal
November 2020
LILI learns a latent embedding of another agent's changing strategy from the ego agent's own observations, plus how that strategy reacts to the ego agent, and trains an RL policy that steers the other agent toward strategies it can exploit. In real robot air hockey it blocks 91% of shots against an opponent that keeps aiming away from it, versus 44% for SAC.
Reactive Human-to-Robot Handovers of Arbitrary ObjectsSeminal
November 2020
A vision-based system lets a Franka arm take unknown objects from a person's hand by segmenting the hand, generating temporally consistent 6-DOF grasps on the remaining points and picking among them with a reactive planner. In a six-person study it took all 10 household test objects from every participant, with an 81.8% average success rate and 1.3 attempts per object.
Physical Interaction as Communication: Learning Robot Objectives Online from Human CorrectionsSeminal
July 2021
Treat a person's push on a robot arm as evidence about the objective the robot should optimize, and update that objective online so the robot changes the rest of its task after the person lets go. In a 10-person study on a 7-DoF arm, learning from corrections cut human effort, interaction time and task cost compared with impedance control (all p<0.0001).
HandoverSim: A Simulation Framework and Benchmark for Human-to-Robot Object HandoversSeminal
May 2022
A PyBullet benchmark for human-to-robot handovers, where a simulated hand replays 1,000 real motion captures and a Panda arm has to take the object without touching the hand. The best of four existing baselines succeeds 64.58% of the time, and the baselines rank the same way they did in a real user study.
Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During DeploymentSeminal
November 2022
Sirius deploys a learned manipulation policy with a human who takes over when it is about to fail, then retrains the policy on that deployment data, upweighting interventions and discarding the robot steps just before them. Autonomous success beats state-of-the-art baselines by 8% in simulation and 27% on a real robot.
Learning Human-to-Robot Handovers from Point CloudsSeminal
March 2023
A point-cloud handover policy is trained in simulation against replayed human motion, first with the hand held still and planner demonstrations, then fine-tuned while the hand moves. It reaches 75% success on the HandoverSim test scenes (69% with the human moving) and beats GA-DDPG on a real Franka (41/60 vs 21/60 handovers for one subject).
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware★ Must read
April 2023
ALOHA is a bimanual teleoperation rig built from off-the-shelf arms for under $20k, and ACT is an imitation learning policy that predicts chunks of future joint targets with a transformer CVAE. From 50 demonstrations per task, ACT slots a battery 96% of the time and opens a translucent condiment cup 84% of the time.
Robots That Ask For Help: Uncertainty Alignment for Large Language Model PlannersSeminal
July 2023
KnowNo frames each step of an LLM planner as a multiple-choice question and uses conformal prediction to keep only the options it cannot rule out. When more than one option remains, the robot asks a human. It meets a user-chosen task success rate with 10-24% less human help than baselines.
AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation SystemSeminal
July 2023
One camera-based teleoperation system that drives different robot arms and dexterous hands, in simulation or the real world, needing only each robot's URDF. On the same xArm6 and Allegro hardware as a robot-specific prior system, it beats that system on 8 of its 10 real-world tasks and ties on the other 2.
GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators★ Must read
September 2023
Build a half-scale, kinematically equivalent copy of a robot arm from 3D-printed parts and hobby servos for under $300, and stream its joint angles straight to the real arm. In a 12-person bimanual user study GELLO averaged 0.92 task success, against 0.72 for VR controllers and 0.63 for 3D mice.
Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation★ Must read
March 2024
H2O trains one RL policy that makes a Unitree H1 humanoid copy a human's whole-body motion in real time from a single RGB camera, after a privileged simulated imitator filters the retargeted AMASS motions down to the ~8.5k of 10k the robot can perform. In simulation it tracks 72.5% of the retargeted motions, versus 67.9% when trained without that filtering.
OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and LearningSeminal
June 2024
OmniH2O trains one whole-body humanoid policy that tracks only head and hand positions, so a Unitree H1 with dexterous hands can be driven by VR, an RGB camera, language or GPT-4o. It tracks 94.1% of 14k motions in simulation (H2O 87.5%), and diffusion policies trained on its 40 minutes of teleoperation data succeed 8/10 on average across four tasks.
Open-TeleVision: Teleoperation with Immersive Active Visual FeedbackSeminal
July 2024
A VR teleoperation system that streams stereo video from an actuated camera on a humanoid's head to the operator's headset, while the robot's head, arms and hands follow the operator's. ACT policies trained on 10-20 of its demonstrations reach 87-100% stage success on four long-horizon tasks on a Unitree H1.
Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation LearningSeminal
July 2024
A real-time system that teleoperates two arms with dexterous hands from Apple Vision Pro hand tracking, with built-in collision and singularity avoidance and $1.2 vibration motors for touch feedback. Policies trained on its demonstrations average a 22% higher success rate than policies trained on AnyTeleop+ data.
Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement LearningSeminal
October 2024
HIL-SERL trains vision-based manipulation policies with RL directly on real robots, starting from 20-30 demonstrations and taking corrections from a human with a SpaceMouse during training. It reaches 100% success on all 13 tasks, against 49.7% for an HG-DAgger baseline given the same human data, with 1.8x faster cycle times.
TWIST: Teleoperated Whole-Body Imitation SystemSeminal
May 2025
A MoCap system streams an operator's whole-body motion to a Unitree G1, and one tracking policy trained with RL plus behavior cloning turns it into balanced skills like crouching to lift a box, kicking a ball and dancing. Adding 150 in-house MoCap clips to training cuts summed tracking error on unseen motions from 2.09 to 1.52.
Recent
Social Humanoid: Socially Responsive Human-Humanoid Interaction
November 2026
A Unitree G1 watches a person through an RGB-D camera and generates a whole-body reaction (handshake, high-five, hug) with a diffusion model over motion tokens shared by humans and the humanoid. On the real robot it succeeds in 90-100% of 30 trials per interaction, with 308 ms median latency.
LumiTac: Compact, High-Rate Six-Axis Optical Tactile Sensing for Human-Robot Interaction
November 2026
A camera-free optical tactile sensor, 12 mm across and under 2 g, that reads three forces and three moments from 20 colour light channels for about USD 7. A small MLP reaches R² of 0.966-0.994 per axis at about 300 Hz, and the sensor guides a real gripper through admittance control.
Correct Robots Before They Make Mistakes: Proactive Human-in-the-Loop Intervention via Preference Learning
November 2026
An XR headset shows the operator the policy's predicted gripper path so they can take over before a mistake, and each rejected prediction is paired with the correction that followed for preference post-training. With five correction episodes, mean real-robot success over HG-DAgger rises from 44.0% to 70.3% for a diffusion policy and from 57.8% to 76.9% for X-VLA.
MABEL: An Open Anthropomorphic Mobile Bimanual Platform for Teleoperation and Learning
November 2026
An open-source 56-DOF mobile bimanual robot with dexterous hands, a pitching torso, a lift and an active neck, for a US$8.1k parts cost. Apple Vision Pro head and hand tracking drives the whole body with 137 ms video latency, and an ACT policy trained through the same controller succeeds in 8 of 10 grasp-and-place trials.
New papers stay for 14 days.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.