Dynamic Human-Robot Role Allocation through Entropy-Based Monitoring and LLM Reasoning

TL;DR The person works on a shared assembly however they like, while a Behavior Tree gives a Franka its next step from multiplied costs (task constraints, distance to the person's hand, and spoken preferences an LLM turns into costs). In a Tangram study, 12 of 14 participants preferred it to fixed task assignment, rating the robot clearer and more predictable, with lower frustration and shorter task and idle times.
What they do
Most role-allocation methods for human-robot collaboration assign tasks to both partners, so the person must follow the plan and often cannot tell why they got a task. Here the person does whichever assembly steps they like and can state preferences by voice. The robot watches what has been done and picks a remaining step that fits the constraints and those preferences. It also fixes pieces the person misplaces.
How it works
- Cost-Based Fallback. At every tick, a Behavior Tree fallback node reorders the robot’s possible activities by the product of several cost vectors and runs the one with the lowest positive cost. A zero in any term blocks an activity, and multiplying avoids tuning weights.
- Cost terms. History (done by the human or the robot), task constraints (robot feasibility, prerequisites, and blocking steps that involve the piece in the person’s hand), and an LLM preference cost from transcribed speech (GPT-5.1, values 0.01/1/100). A distance term favors pieces far from the person’s hand.
- Activity monitoring. SAM 2 tracks the hand and pieces from a top-down camera. Mutual information and entropy of hand-object and object-object distances turn each action into scene graphs, which are matched against reference graphs to detect completed or misplaced steps.
Results
- In a pilot with one expert user, the robot followed two spoken preferences, flagged a misplaced piece and put it back correctly.
- With 14 participants, the proposed condition scored higher on clarity (p<0.05) and predictability (p<0.01) and lower on frustration (p<0.01). The other NASA-TLX scores showed no significant difference.
- The LLM interpreted 100% of requests correctly. Under assigned tasks, only 31% of participants identified what drove allocation, and 38% said they could not understand it.
- Execution and idle times were significantly lower (p<0.001), and 12 of 14 participants preferred the proposed condition.
Limitations
For the 14-person study, SAM tracking was too slow, so only the hand was tracked online, piece poses were preset, and the experimenter recognized activities by hand. The LLM prompts are tailored to easily named Tangram pieces. Requests made during a robot action wait until the next allocation, and motion planning ignores the person’s motion.
Videos
Figures




Abstract
Role allocation strategies in human-robot collaboration typically assign tasks to both the human and the robot, limiting the user's ability to take initiative. Furthermore, allocation decisions may conflict with the user's intentions, and the rationale behind such allocations may remain unclear, potentially reducing both acceptance and collaboration fluency. The novelty of the proposed framework is that the human is left free to contribute to the task, while the robot is dynamically allocated activities through a cost-driven Behavior Tree architecture according to user-expressed preferences and behavior. Explicit preferences are extracted from natural language instructions using a large language model, whereas human activity is monitored online through a Shannon entropy-based analysis of hand-object interactions. Furthermore, the framework can recognize anomalies in human execution and autonomously trigger corrective robot actions. The approach was validated through a pilot study and a multi-subject experiment involving 14 participants. Results demonstrate the potential of such an allocation strategy for enabling intuitive and engaging collaboration without increasing the perceived workload.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
