← Paper Digest

SGIP: Learning Scene-Grounded Interaction Priors for Scene-Aware Human Motion Prediction

Donghyeon Soon, Jiil Yoon, Hyeondong Kim, Daehee Park

November 2026

Kitchen scene, gaze-free SIF3D (left) vs SGIP (right) forecasting the same observed motion.

TL;DR SGIP first predicts which object or open region of a 3D scene a person is heading for, then forecasts their trajectory and full-body pose toward it, using an LLM-guided teacher only at training time. Without gaze or intent at inference, it cuts GIMO trajectory error to 549 mm from 642 mm for the best comparable baseline.

What they do

In a cluttered room the same observed approach could end at one of several chairs, at the sofa, or nowhere in particular. Earlier scene-aware predictors leave this choice implicit or use gaze, which needs extra sensing. SGIP predicts the destination explicitly, as a distribution over scene tokens (object instances plus one free-space token) computed from the observed body motion and the scene point cloud alone, and conditions the trajectory and pose forecast on the most likely token.

How it works

  • Candidates. OpenIns3D zero-shot instance segmentation of the scene, pruned to objects inside a 120° cone in front of the observed head and torso, plus the navigation token.
  • Teacher (training only). A VLM labels each training clip with a textual intent (“take some rest”); an LLM scores how well each candidate supports it, and a geometric score from the approach direction picks among same-class instances.
  • Student (deployed). A permutation-equivariant MLP reproduces the teacher’s prior from motion and scene only, trained with a KL loss on the prior plus an L1 loss toward the teacher’s trajectory and pose.
  • Predictor. Trajectory first, then full-body pose, both conditioned on the chosen token’s embedding and mask. Decoding the top-K tokens separately gives K alternative futures.

Results

  • Best of all methods without gaze or intent on all four metrics on GIMO and PROX: GIMO path error 549 mm vs 642 mm, PROX destination error 576 vs 659 mm.
  • Decoding the top 3 tokens lowers minADE/minFDE from 0.549/0.681 to 0.467/0.523 m.
  • The student ranks the annotated target first in 76% of GIMO sequences (teacher 86%; LLM or geometry score alone 54% and 51%). Prior distillation gives the largest gain, from 740/838 to 574/688 mm path/destination error.
  • 132.8 ms per observation on one RTX 3090, after 61.7 s of one-time scene preprocessing.

Limitations

Accuracy depends on the scene segmentation: merging or splitting the target object raises destination error to 875-908 mm. Every new scene needs segmentation first, and other people are treated as static tokens, so on the multi-person HOI-M3 dataset its path error is higher than HUMOF’s (291 vs 254 mm).

Videos

Seminar room, gaze-free SIF3D (left) vs SGIP (right) forecasting the same observed motion.

Figures

From Figure 1: (a) Previous methods rely on motion-scene distance or gaze without an explicit interaction target. (b)...
From Figure 1: (a) Previous methods rely on motion-scene distance or gaze without an explicit interaction target. (b) SGIP predicts a distribution over candidate scene tokens, covering object instances and free space, from the observed motion and 3D scene alone. Here it puts the highest probability on the sofa, which conditions the trajectory and pose forecast of walking to and sitting on it.
From Figure 2: Pipeline. A training-only teacher (top) fuses an LLM rubric score with a geometric compatibility score...
From Figure 2: Pipeline. A training-only teacher (top) fuses an LLM rubric score with a geometric compatibility score into a prior over grounding tokens. The deployed student (bottom) predicts that prior from motion and scene alone with a permutation-equivariant MLP, trained by prior and motion distillation. The selected token conditions the predictor of future trajectory and full-body pose (right).
From Figure 4: SGIP (top rows) vs the gaze-free baseline SIF3D (bottom rows) in three GIMO scenes, showing the top-do...
From Figure 4: SGIP (top rows) vs the gaze-free baseline SIF3D (bottom rows) in three GIMO scenes, showing the top-down trajectory, the final interaction frame and the pose sequence. Yellow: observed motion; red: ground-truth future; blue: prediction.
Abstract

Human motion prediction (HMP) aims to forecast future movement from an observed motion sequence, and recent methods have incorporated 3D scene geometry and object semantics to improve long-term plausibility. However, the destination of future motion remains ambiguous in cluttered scenes: the same observed motion can lead to different interaction targets or navigation modes. Existing methods either handle this ambiguity implicitly or rely on auxiliary signals like gaze that are hard to obtain in practice. To address this, we propose SGIP, which explicitly predicts where in the scene future motion will be grounded across both objects and open navigation regions, and uses it to condition future trajectory and full-body pose prediction. This scene-grounded interaction prior is learned via intent distillation: at training time only, a teacher model scores candidate scene regions by combining an LLM prompted with a textual description of the person's intent with geometric motion-scene features. A student model learns to reproduce these scores from only the observed motion and 3D scene, making intent and LLM unnecessary at test time. Across multiple benchmark datasets, SGIP outperforms prior scene-aware baselines on both trajectory and pose accuracy in a realistic deployment setting where neither gaze nor intent is available at inference, demonstrating the effectiveness of the proposed framework. The project page is available at https://sgip2026.github.io.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.