Shared Autonomy with Learned Latent Actions
TL;DR A joystick-to-arm decoder conditioned on the robot's belief over goals is combined with shared autonomy, so the same low-DoF inputs first steer toward a goal and then control fine manipulation there. In a 10-person eating study with a 7-DoF arm, this combination had the lowest total and idle time on both tasks (p<.05).
What they do
Eating with an assistive arm takes coarse reaching (which plate) and precise manipulation at that goal (cutting, stabbing, scooping), all through a low-dimensional joystick. Shared autonomy helps reach discrete goals but not afterwards; learned latent actions give intuitive low-DoF control but no guidance. The paper combines them, so the meaning of the user’s input changes with the robot’s confidence in the goal.
How it works
- Belief-conditioned latent actions. An autoencoder trained on kinesthetic demonstrations learns a decoder phi(z, s, b) from input z, robot state s and goal belief b to a joint action. While the belief is uncertain, z selects the goal; once the robot is confident, z controls the manipulation.
- Shared autonomy. The decoded action is blended with a robot action that pulls toward each candidate goal in proportion to its belief. A Lyapunov argument bounds the robot to a ball around the true goal that shrinks as confidence grows.
- Entropy term. Training also maximizes the entropy over which goals the latent actions can reach from each (s, b), so users are not trapped at a wrong goal.
- New goals. Contexts near an unseen goal are mapped to the equivalent context at a known goal, decoded there and mapped back, without retraining.
Results
- Simulated users (1-DoF input, two cups): shared autonomy lowered time and final error at every rationality level. With the 7-DoF Franka, LA took 45% more time than LA+SA at beta=75 and 30% more at beta=1000.
- Users who changed their mind stayed stuck at the wrong goal with LA+SA once the switch came late; with LA+SA+Entropy they could switch goals. For new cups, grasp error grew linearly with distance from the training cup (y ≈ 0.11x, R²=0.98), versus y = x without the mapping.
- User study (10 participants, 2-DoF joystick, 7-DoF arm, 20 min of demonstrations): shared autonomy lowered goal error (F(1,18)=29.9, p<.001). Users of direct retargeting stayed in a stabbing motion, while latent actions let them switch between stabbing, scooping and dipping. LA+SA had the lowest total and idle time (p<.05).
Limitations
The 10 participants were able-bodied and had equal, brief practice with each condition; daily users’ greater experience could bias them toward familiar interfaces, and the target population will likely give noisier inputs. Goals form a discrete set known in advance.
Videos
Figures
Abstract
Assistive robots enable people with disabilities to conduct everyday tasks on their own. However, these tasks can be complex, containing both coarse reaching motions and fine-grained manipulation. For example, when eating, not only does one need to move to the correct food item, but they must also precisely manipulate the food in different ways (e.g., cutting, stabbing, scooping). Shared autonomy methods make robot teleoperation safer and more precise by arbitrating user inputs with robot controls. However, these works have focused mainly on the high-level task of reaching a goal from a discrete set, while largely ignoring manipulation of objects at that goal. Meanwhile, dimensionality reduction techniques for teleoperation map useful high-dimensional robot actions into an intuitive low-dimensional controller, but it is unclear if these methods can achieve the requisite precision for tasks like eating. Our insight is that---by combining intuitive embeddings from learned latent actions with robotic assistance from shared autonomy---we can enable precise assistive manipulation. In this work, we adopt learned latent actions for shared autonomy by proposing a new model structure that changes the meaning of the human's input based on the robot's confidence of the goal. We show convergence bounds on the robot's distance to the most likely goal, and develop a training procedure to learn a controller that is able to move between goals even in the presence of shared autonomy. We evaluate our method in simulations and an eating user study. See videos of our experiments here: https://youtu.be/7BouKojzVyk
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.