Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

HEIR: Harness Egocentric Intent for Human-Robot Interactions

Yichi Zhang, Bo Wang, Zhenghan Zhu, Yutong Han, Jiayin Zou, Gao Kejun, Chang Liu, Shaoxuan Xie, Chi Zhang, Shi Haoyang, Zongxuan Lai, Sen Cui, Guocai Yao, Miao Liu

November 2026

A recorded session at 5x speed from the participant's eye-tracking glasses, the robot's camera and two external cameras as the robot moves through the room.

TL;DR HEIR is a 38.6-hour dataset of household requests that pairs a person's speech, egocentric video and gaze with a teleoperated mobile dual-arm robot's observations and navigation-manipulation trajectories. Adding the human view and gaze raises a VLM's object-location accuracy from 55% (robot view only) to 78%, yet a real-robot system built from the benchmarked models completes only 18.5% of requests end to end.

What they do

A home robot has to work out what a person means (“bring me that thing”) and then carry it out over several navigation and manipulation steps. HEIR is a dataset for this. A participant wearing eye-tracking glasses makes spoken requests while an operator teleoperates a Galaxea R1 Lite mobile dual-arm robot to fulfil them. The authors benchmark intent inference, subtask planning, navigation and manipulation on it, offline and on the real robot.

How it works

  • Recording. Egocentric video, gaze and speech are synchronized with four robot camera streams and arm, gripper, torso and base states and actions. Requests follow one another in a changing scene, so later ones can refer to earlier ones.
  • Annotation. Humans label each request’s intent (task, objects, goal, nominal plan), whether the target was visible to the robot, and the subtask boundaries. GPT-5.6 adds subgoals and completion conditions, and humans review every episode.
  • Scale. 17 participants, 300 sessions, 38.56 h, 1,836 requests and 6,416 subtasks, with 60 sessions held out for testing.
  • Deployed system. A VLM intent reasoner, a planner choosing Continue, Switch or Done, a VLN model for navigation and a VLA for manipulation.

Results

  • Intent inference with Gemma-4-31B: object-location accuracy is 20.5% from text alone, 54.9% with the robot view, 71.1% with the human view added and 77.7% with gaze. Plan exact match rises from 30.3% to 78.1%.
  • Fine-tuning on HEIR raises planner decision accuracy by 25-37 points and InternVLA-N1’s navigation action accuracy from 49.1% to 90.7%.
  • On 20 real pick-and-place tasks (3 trials each), Lingbot-VLA2, FastWAM and π0.5 succeed 68.3%, 60.0% and 36.7% of the time, matching their offline ranking.
  • The full system completes 81.25% of 32 navigation subtasks but only 18.52% of 27 end-to-end requests, which the authors attribute to errors accumulating across components.

Limitations

All sessions come from one staged room, so the test split varies furniture and object placement, not the environment. The real-robot benefit of gaze is shown only in a few qualitative cases, and some planning metrics rely on LLM judges.

Videos

A bed-making scene at 5x speed from the robot's wrist and top cameras, the participant's gaze-tracked view and an external camera.

Figures

From Figure 2: (a) Hardware: Pupil Labs Neon glasses record the participant's egocentric view, gaze and audio, and th...
From Figure 2: (a) Hardware: Pupil Labs Neon glasses record the participant's egocentric view, gaze and audio, and the mobile robot has head stereo and wrist cameras, 6-DoF arms, a 3-DoF torso and steering wheels. (b) A session holds several episodes, each a natural request such as "Give me that folder." carried out as navigation and manipulation subtasks. (c) Humans annotate the instruction and subtask timing, then a VLM adds a description of each subtask.
From Figure 3: HEIR has 300 sessions across 20 task categories in home, work and kitchen scenarios, totaling 38.56 ho...
From Figure 3: HEIR has 300 sessions across 20 task categories in home, work and kitchen scenarios, totaling 38.56 hours, 1,836 episodes and 6,416 subtasks, shown with duration distributions, the train/test split and participant statistics.
From Figure 5: The real-robot pipeline. For the request
From Figure 5: The real-robot pipeline. For the request "Bring me that folder.", the intent reasoner infers a structured intent from speech, the human view with gaze and the robot view, and a closed-loop planner dispatches subtasks to the navigation and manipulation backends until Done.
Abstract

For robots to assist, serve, and collaborate with humans, they must not only interpret human requests conveyed through language and multimodal intentional cues, but also execute long-horizon tasks that interleave navigation and manipulation. In this paper, we introduce HEIR, a resource for studying how to Harness Egocentric Intent for Human-Robot Interaction. HEIR pairs human speech, measured gaze, and egocentric video with robot observations and teleoperated navigation-manipulation trajectories, while each session preserves successive episodes, with each request fulfilled through multiple execution subtasks. The captured data, together with human-authored annotations, support the study of intent grounding, execution planning, and task completion within a unified interaction setting. To demonstrate the value of HEIR, we conduct extensive offline experiments to evaluate how models benefit from additional human-centric context across these tasks, together with real-robot experiments that assess execution task completion. HEIR thus provides a valuable resource for studying user-centric robot interaction in naturalistic household environments. Our demo and dataset are available online.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.