SGIP: Learning Scene-Grounded Interaction Priors for Scene-Aware Human Motion Prediction
TL;DR SGIP first predicts which object or open region of a 3D scene a person is heading for, then forecasts their trajectory and full-body pose toward it, using an LLM-guided teacher only at training time. Without gaze or intent at inference, it cuts GIMO trajectory error to 549 mm from 642 mm for the best comparable baseline.
What they do
In a cluttered room the same observed approach could end at one of several chairs, at the sofa, or nowhere in particular. Earlier scene-aware predictors leave this choice implicit or use gaze, which needs extra sensing. SGIP predicts the destination explicitly, as a distribution over scene tokens (object instances plus one free-space token) computed from the observed body motion and the scene point cloud alone, and conditions the trajectory and pose forecast on the most likely token.
How it works
- Candidates. OpenIns3D zero-shot instance segmentation of the scene, pruned to objects inside a 120° cone in front of the observed head and torso, plus the navigation token.
- Teacher (training only). A VLM labels each training clip with a textual intent (“take some rest”); an LLM scores how well each candidate supports it, and a geometric score from the approach direction picks among same-class instances.
- Student (deployed). A permutation-equivariant MLP reproduces the teacher’s prior from motion and scene only, trained with a KL loss on the prior plus an L1 loss toward the teacher’s trajectory and pose.
- Predictor. Trajectory first, then full-body pose, both conditioned on the chosen token’s embedding and mask. Decoding the top-K tokens separately gives K alternative futures.
Results
- Best of all methods without gaze or intent on all four metrics on GIMO and PROX: GIMO path error 549 mm vs 642 mm, PROX destination error 576 vs 659 mm.
- Decoding the top 3 tokens lowers minADE/minFDE from 0.549/0.681 to 0.467/0.523 m.
- The student ranks the annotated target first in 76% of GIMO sequences (teacher 86%; LLM or geometry score alone 54% and 51%). Prior distillation gives the largest gain, from 740/838 to 574/688 mm path/destination error.
- 132.8 ms per observation on one RTX 3090, after 61.7 s of one-time scene preprocessing.
Limitations
Accuracy depends on the scene segmentation: merging or splitting the target object raises destination error to 875-908 mm. Every new scene needs segmentation first, and other people are treated as static tokens, so on the multi-person HOI-M3 dataset its path error is higher than HUMOF’s (291 vs 254 mm).
Videos
Figures



Abstract
Human motion prediction (HMP) aims to forecast future movement from an observed motion sequence, and recent methods have incorporated 3D scene geometry and object semantics to improve long-term plausibility. However, the destination of future motion remains ambiguous in cluttered scenes: the same observed motion can lead to different interaction targets or navigation modes. Existing methods either handle this ambiguity implicitly or rely on auxiliary signals like gaze that are hard to obtain in practice. To address this, we propose SGIP, which explicitly predicts where in the scene future motion will be grounded across both objects and open navigation regions, and uses it to condition future trajectory and full-body pose prediction. This scene-grounded interaction prior is learned via intent distillation: at training time only, a teacher model scores candidate scene regions by combining an LLM prompted with a textual description of the person's intent with geometric motion-scene features. A student model learns to reproduce these scores from only the observed motion and 3D scene, making intent and LLM unnecessary at test time. Across multiple benchmark datasets, SGIP outperforms prior scene-aware baselines on both trajectory and pose accuracy in a realistic deployment setting where neither gaze nor intent is available at inference, demonstrating the effectiveness of the proposed framework. The project page is available at https://sgip2026.github.io.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
