Rhythmic-Semantic Co-Speech Gestures for Emotional Dialogue in Robots

TL;DR A text-only system that gives the social robot Haru both iconic, emotion-carrying gestures and rhythmic beat gestures while it speaks, using small BERT-style models trained on BEAT2. In an online study with 60 participants it scored highest on speech-gesture correlation and was ranked best most often (40% of trials).
What they do
Robot co-speech gesture systems usually produce either rhythmic beat motion or iconic gestures that carry meaning, rarely both. This paper generates both in one stream for the non-humanoid social robot Haru, conditioned on the utterance and one of four emotions (joy, anger, sadness, fear). It uses text only, so the robot does not have to wait for synthesized speech audio, which the authors note can take about 2 s with expressive TTS.
How it works
- Iconic Module. A BERT classifier decides whether a sentence gets an iconic gesture, and a BERT regressor predicts its placement (the peak word) and intensity. Both use about 17,000 BEAT2 sentences, split 80/20 into training and test.
- Beat Module. BEAT2 has no beat-intensity labels, so the authors derive them from head and arm motion energy, discretized into low, mid and high. A RoBERTa classifier predicts this level per sentence from the text, the emotion, the iconic output and timing estimated from the text.
- Robot mapping. Iconic predictions select from a library of hand-made Haru animations, three intensity levels per emotion. Beats fill 0.8 s windows outside iconic spans using ten head, eye and base motion templates scaled by intensity.
Results
- Iconic activation: BERT reaches 0.61 accuracy in 3.7 ms, versus 0.51 accuracy in 596 ms for a GPT-4o prompt. RoBERTa predicts beat intensity with 0.40 accuracy across three classes.
- In a within-subjects video study (N=60, 15 per emotion), the system scored significantly higher on speech-gesture correlation than the LLM and random baselines, with a trend over the ground-truth BEAT2 labels.
- No significant differences in anthropomorphism or likeability.
- Participants ranked it best in 40% of trials, against 21.7% each for the LLM and ground-truth baselines (Figure 4).
Limitations
Text-only inference ignores prosody, beat intensity is coarse, and iconic gestures come from a fixed library. The study used online videos rather than live interaction, tested only Haru, and did not separate the effects of iconic and beat motion.
Videos
Figures




Abstract
Co-speech gestures can increase engagement and improve speech understanding. Yet, most data-driven robot systems generate either rhythmic beat-like motion or semantically meaningful iconic gestures, but not both. We present an emotion-aware rhythmic-semantic co-speech gesture system that generates iconic and beat behaviors in real time from text and the robot's internal emotional state. We formalize the task of beat-iconic gesture generation using emotion and propose lightweight models that predict iconic timing and intensity, then fill the remaining spans with emotion-conditioned beats at an appropriate intensity level. We train on the BEAT2 dataset and deploy the system on the social robot Haru. In an online user study with 60 participants, our method shows stronger speech-gesture alignment and is preferred over baseline methods.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
