Feel the Bite: Robot-Assisted Inside-Mouth Bite Transfer using Robust Mouth Perception and Physical Interaction-Aware Control

Rajat Kumar Jenamani, Daniel Stabile, Ziang Liu, Abrar Anwar, Katherine Dimitropoulou, Tapomayukh Bhattacharjee

Seminal March 2024

Feeding a care recipient with schizencephaly quadriplegia, tongue spasticity and gag and bite reflexes.

TL;DR A feeding robot that places food inside the mouth. It tracks the mouth through utensil occlusion with two wrist cameras and picks a compliant controller from a visual-plus-haptic classification of each contact. The system fed 13 care recipients with severe mobility limitations; 9 of them could not lean forward for a standard outside-mouth bite.

What they do

Most feeding robots stop in front of the mouth, but many people with severe mobility limitations cannot lean forward to take the bite. This system places food inside the mouth. It keeps tracking the mouth while the utensil hides it, and tells apart incidental bumps, spasms, tongue pushes that guide the utensil, and bites.

How it works

Hardware. A Kinova Gen3 arm holds a feeding tool. Two RealSense cameras sit on the wrist, one above and one below the utensil, and an ATI F/T sensor measures contact forces.
FEMTO mouth perception. Two frozen DECA encoders, one per camera, feed fully-connected layers that predict FLAME head parameters. The network is finetuned on about 5000 unoccluded images with a utensil painted in, using DECA’s predictions on the clean images as labels. The personalized head model is projected to keypoints and registered to depth from the unoccluded parts of the face. The pipeline runs at 5-10 Hz.
Interaction-aware control. An SVM classifies a 100 ms window of face keypoints and haptic features, trained on 3072 recorded interactions. Spasms and tongue pushes switch the arm to a force-tracking controller. A goal-tracking controller moves the food to its target and withdraws once a bite is detected.

Results

Mouth keypoint RMSE under synthetic occlusion: 3.69 mm for FEMTO, against 4.72 mm for DECA and 83-110 mm for earlier feeding-system methods.
Contact classification F1: 0.903 with both modalities, against 0.857 haptic-only and 0.860 visual-only. It is 0.872 on unseen participants, and 0.719 zero-shot on later study data, rising to 0.772 after per-person finetuning.
In ablations with 15 and 14 participants without impairments, continuous tracking beat one-time mouth estimation in all three head-movement scenarios. In the spasm scenario, full four-way classification was rated highest and treating every contact as a bite was rated lowest.
The system fed 13 care recipients at three sites. Of the 4 who could also use outside-mouth transfer, some preferred inside-mouth because it took less effort.

Limitations

Users’ needs differ a lot: participants with small mouth openings found the spoon too big, and preferred speeds varied. The classifier needs per-person finetuning to cope with new users. The care-recipient study covered short sessions only.

Videos

Inside-mouth feeding of a care recipient with a C4-C5 spinal cord injury, 4x speed.

Feeding a care recipient with arthrogryposis, tongue and masseter spasticity and a gag reflex.

Figures

From Figure 1: The inside-mouth bite transfer system and its two components: multi-view mouth perception, and interac... — From Figure 1: The inside-mouth bite transfer system and its two components: multi-view mouth perception, and interaction-aware control that classifies a contact as incidental, bite, impulsive or in-mouth manipulation and switches between goal-tracking and force-tracking controllers.

From Figure 2: Self-supervised multi-view training of FEMTO on images with a synthetically added utensil (left), its ... — From Figure 2: Self-supervised multi-view training of FEMTO on images with a synthetically added utensil (left), its predictions under occlusion compared with DECA (center), and the pipeline that registers the reconstructed head model to 3D keypoints to get the mouth pose (right).

From Figure 6: F1 score of the SVM contact classifier on a novel participant as more of that participant's data is us... — From Figure 6: F1 score of the SVM contact classifier on a novel participant as more of that participant's data is used for finetuning, for visual, haptic and combined inputs.

From Figure 7: The system feeding a care recipient at home (left); participants with limited mouth opening, a left-mo... — From Figure 7: The system feeding a care recipient at home (left); participants with limited mouth opening, a left-molars-only bite, weak jaw control and spasms (center); care recipients' ratings of safety, comfort and technology acceptance (right).

Abstract

Robot-assisted feeding can greatly enhance the lives of those with mobility limitations. Modern feeding systems can pick up and position food in front of a care recipient's mouth for a bite. However, many with severe mobility constraints cannot lean forward and need direct inside-mouth food placement. This demands precision, especially for those with restricted mouth openings, and appropriately reacting to various physical interactions - incidental contacts as the utensil moves inside, impulsive contacts due to sudden muscle spasms, deliberate tongue maneuvers by the person being fed to guide the utensil, and intentional bites. In this paper, we propose an inside-mouth bite transfer system that addresses these challenges with two key components: a multi-view mouth perception pipeline robust to tool occlusion, and a control mechanism that employs multimodal time-series classification to discern and react to different physical interactions. We demonstrate the efficacy of these individual components through two ablation studies. In a full system evaluation, our system successfully fed 13 care recipients with diverse mobility challenges. Participants consistently emphasized the comfort and safety of our inside-mouth bite transfer system, and gave it high technology acceptance ratings - underscoring its transformative potential in real-world scenarios. Supplementary materials and videos can be found at http://emprise.cs.cornell.edu/bitetransfer/ .

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong