← Paper Digest

Dynamic Human-Robot Role Allocation through Entropy-Based Monitoring and LLM Reasoning

Elena Merlo, Marta Lagomarsino, Hamidreza Raei, Arash Ajoudani

November 2026

TL;DR The person works on a shared assembly however they like, while a Behavior Tree gives a Franka its next step from multiplied costs (task constraints, distance to the person's hand, and spoken preferences an LLM turns into costs). In a Tangram study, 12 of 14 participants preferred it to fixed task assignment, rating the robot clearer and more predictable, with lower frustration and shorter task and idle times.

What they do

Most role-allocation methods for human-robot collaboration assign tasks to both partners, so the person must follow the plan and often cannot tell why they got a task. Here the person does whichever assembly steps they like and can state preferences by voice. The robot watches what has been done and picks a remaining step that fits the constraints and those preferences. It also fixes pieces the person misplaces.

How it works

  • Cost-Based Fallback. At every tick, a Behavior Tree fallback node reorders the robot’s possible activities by the product of several cost vectors and runs the one with the lowest positive cost. A zero in any term blocks an activity, and multiplying avoids tuning weights.
  • Cost terms. History (done by the human or the robot), task constraints (robot feasibility, prerequisites, and blocking steps that involve the piece in the person’s hand), and an LLM preference cost from transcribed speech (GPT-5.1, values 0.01/1/100). A distance term favors pieces far from the person’s hand.
  • Activity monitoring. SAM 2 tracks the hand and pieces from a top-down camera. Mutual information and entropy of hand-object and object-object distances turn each action into scene graphs, which are matched against reference graphs to detect completed or misplaced steps.

Results

  • In a pilot with one expert user, the robot followed two spoken preferences, flagged a misplaced piece and put it back correctly.
  • With 14 participants, the proposed condition scored higher on clarity (p<0.05) and predictability (p<0.01) and lower on frustration (p<0.01). The other NASA-TLX scores showed no significant difference.
  • The LLM interpreted 100% of requests correctly. Under assigned tasks, only 31% of participants identified what drove allocation, and 38% said they could not understand it.
  • Execution and idle times were significantly lower (p<0.001), and 12 of 14 participants preferred the proposed condition.

Limitations

For the 14-person study, SAM tracking was too slow, so only the hand was tracked online, piece poses were preset, and the experimenter recognized activities by hand. The LLM prompts are tailored to easily named Tangram pieces. Requests made during a robot action wait until the next allocation, and motion planning ignores the person’s motion.

Videos

Full video (YouTube)

Figures

From Figure 1: LLM-interpreted user preferences, human factor monitoring, human activity identification and task cons...
From Figure 1: LLM-interpreted user preferences, human factor monitoring, human activity identification and task constraints form a cost vector that a Cost-Based Fallback node in a Behavior Tree uses to choose the robot's next activity while the user acts freely. Orange modules set which activities are available, yellow modules which to prioritize.
From Figure 2: How the cost vectors combine to pick the robot's activity, before (left) and after (right) the spoken ...
From Figure 2: How the cost vectors combine to pick the robot's activity, before (left) and after (right) the spoken request "Fill drawer A". The activity in green goes to the top of the fallback node's children, those in red to the bottom.
From Figure 3: The experimental setup, with the participant facing the Franka arm across the Tangram pieces, a top-do...
From Figure 3: The experimental setup, with the participant facing the Franka arm across the Tangram pieces, a top-down camera, a microphone and a GUI showing the target figure and the robot's current action.
From Figure 6: User study, proposed allocation (A) versus assigned tasks (B). Left: questionnaire ratings, with aster...
From Figure 6: User study, proposed allocation (A) versus assigned tasks (B). Left: questionnaire ratings, with asterisks marking significant differences in paired Wilcoxon signed-rank tests. Middle: NASA-TLX workload scores. Right: task execution time and human idle time.
Abstract

Role allocation strategies in human-robot collaboration typically assign tasks to both the human and the robot, limiting the user's ability to take initiative. Furthermore, allocation decisions may conflict with the user's intentions, and the rationale behind such allocations may remain unclear, potentially reducing both acceptance and collaboration fluency. The novelty of the proposed framework is that the human is left free to contribute to the task, while the robot is dynamically allocated activities through a cost-driven Behavior Tree architecture according to user-expressed preferences and behavior. Explicit preferences are extracted from natural language instructions using a large language model, whereas human activity is monitored online through a Shannon entropy-based analysis of hand-object interactions. Furthermore, the framework can recognize anomalies in human execution and autonomously trigger corrective robot actions. The approach was validated through a pilot study and a multi-subject experiment involving 14 participants. Results demonstrate the potential of such an allocation strategy for enabling intuitive and engaging collaboration without increasing the perceived workload.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.