Towards Robotic Feeding: Role of Haptics in Fork-based Food Manipulation

Tapomayukh Bhattacharjee, Gilwoo Lee, Hanjun Song, Siddhartha S. Srinivasa

Seminal April 2018

TL;DR They recorded 3304 trials of 12 people feeding a mannequin with a force/torque-sensing fork, built a taxonomy of feeding strategies, and classified food into four compliance categories from haptic and motion signals, with a TCN reaching 80.47% accuracy. A robot using one fixed vertical skewering motion acquired bites less reliably than people.

What they do

Feeding someone with a fork means handling food of very different compliance, size and shape. The authors instrument a fork, record 12 people feeding 12 food items (plus noodles and potato salad) to a mannequin, and analyze the forces and motions they use. From this they build a taxonomy of feeding strategies, train classifiers that sort food into four compliance categories from haptic and motion signals, and compare people with a robot that uses one fixed skewering motion.

How it works

Forque. A fork-shaped tool with an embedded ATI Nano25 6-axis force/torque sensor and mocap markers tracked by 6 Optitrack cameras at 120 FPS; RGBD and DSLR cameras record the scene. 3304 trials, more than 18 hours of data.
Taxonomy. Feeding splits into rest, approach, bite acquisition and transport. Subjects wiggled the fork to pierce hard and hard-skin items, tilted it so soft items like banana would not slip off, used the plate edge to stabilize rolling items, and oriented the fork during transport so the food was easy to bite.
Classification. LSTM, TCN, SVM and per-category HMMs label each skewering as hard-skin, hard, medium or soft from forces, torques, Forque pose and their derivatives.
Robot. A Fetch arm holding the Forque replays demonstrated waypoints with a vertical skewer, 240 trials.

Results

TCN is best at 80.47 ± 1.17% (3-fold cross-validation); the force along the fork axis alone gets 74.22 ± 0.29%. Errors are mostly between neighbouring categories. Classification uses only the first 0.82 s of skewering.
A TCN trained on robot data reaches 83.1 ± 4.8%, but the human-trained classifier gets only 50.6 ± 3.7% on robot data because the forces differ.
People found hard-skin items hardest; the robot struggled with both hard-skin and soft items, and its success rate was significantly below the humans’ for hard-skin, medium and soft (p < 0.0025).

Limitations

The robot uses one fixed position-controlled skewer without scooping or twirling; compliance-dependent force control is left to future work. Food is fed to a mannequin, and interaction with an eater, such as bite timing, is out of scope. Even after learning from failures, people were never perfect at manipulating items of varying compliance.

Figures

From Figure 1: A Fetch robot feeds a mannequin with the Forque, in the same setup used for the human study.

From Figure 3: A partial taxonomy of feeding strategies across the rest, approach, bite acquisition and transport pha... — From Figure 3: A partial taxonomy of feeding strategies across the rest, approach, bite acquisition and transport phases: skewer (tilt, wiggle, partial), scoop and twirl during bite acquisition, and tension or compression on the fork during transport.

From Figure 6: Confusion matrix of the TCN on human data over the four compliance categories; most errors fall betwee... — From Figure 6: Confusion matrix of the TCN on human data over the four compliance categories; most errors fall between neighbouring categories.

From Figure 7: Bite acquisition success rate of humans and of the position-controlled robot for each compliance categ... — From Figure 7: Bite acquisition success rate of humans and of the position-controlled robot for each compliance category.

Abstract

Autonomous feeding is challenging because it requires manipulation of food items with various compliance, sizes, and shapes. To understand how humans manipulate food items during feeding and to explore ways to adapt their strategies to robots, we collected a rich dataset of human trajectories by asking them to pick up food and feed it to a mannequin. From the analysis of the collected haptic and motion signals, we demonstrate that humans adapt their control policies to accommodate to the compliance and shape of the food item being acquired. We propose a taxonomy of manipulation strategies for feeding to highlight such policies. As a first step to generate compliance-dependent policies, we propose a set of classifiers for compliance-based food categorization from haptic and motion signals. We compare these human manipulation strategies with fixed position-control policies via a robot. Our analysis of success and failure cases of human and robot policies further highlights the importance of adapting the policy to the compliance of a food item.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong