A computational framework for understanding embodied plans

Aalap D. Shah, Basil Saeed, Joshua B. Tenenbaum, Tobias Gerstenberg, Hyowon Gweon, Ilker Yildirim

November 2026

TL;DR GenEA models how people infer others' goals and plans by adding noisy kinematics and a physics-simulation level to task-and-motion planning, then turning plan costs into a probability distribution over goals and plans. Its predictions correlate with human judgments at r = .88 to .93 across three preregistered experiments, ahead of plain TAMP, heuristics and a vision-language model.

What they do

People read goals and plans off the bodily actions of others. Existing computational accounts treat the agent as a point in a 2D grid world. GenEA (Generative Embodied Agent) is an agent model with an articulated body that manipulates blocks and walks or jumps across platforms. It is scored on how well it predicts human judgments about such scenes, not on robot task success.

How it works

Three levels. A symbolic level holds action primitives (Pick, Cache, Fix; Step, Jump, Turn) and an automaton over them. A kinematics level turns each into joint trajectories with noisy endpoints, using KOMO for manipulation and Crocoddyl with the lower body of a Talos humanoid for navigation. A dynamics level, absent from standard TAMP, runs a physics engine (PhysX or MuJoCo) so objects fall or topple under gravity.
Distribution over plans. A breadth-first planner returns the plans that reach a final configuration. Each plan gets a Boltzmann probability from its cost, the kinetic energy of its trajectories.
Queries. Goal inference, action inference, one hand versus two, and task difficulty are all read out of this joint distribution over goals and plans, with a parameter-free choice rule.

Results

Goal and action inference on block structures (60 participants): r = .88 with human ratings, significantly above TAMP without the dynamics level, geometric heuristics and a vision-language model.
One hand or two (22 trials, 60 participants): r = .93.
Difficulty of obstacle courses (22 trials, 60 participants): r = .91. Humans and GenEA both rate wooden platforms harder than stone ones; TAMP shows no such sensitivity.
A GenEA-derived evidence rate also correlates with human response times (r = -.54 and r = -.59).

Limitations

GenEA is not image-computable: it is handed the object configurations, and the authors name perception as the extension needed for robotics use. Action primitives, their rules and the noise parameters are predefined by hand. The authors frame it as a computational-level account, not the algorithm the mind runs. Stimuli are small block scenes and platform courses.

Figures

From Figure 2: Overview of GenEA. (a) A multilevel state-space couples the coarse biomechanics of an agent with nearb... — From Figure 2: Overview of GenEA. (a) A multilevel state-space couples the coarse biomechanics of an agent with nearby objects. The symbolic level defines an automaton of action primitives for the body (such as picking up an object or jumping across platforms), the kinematics level captures the noisy motion trajectories of the agent’s end-effectors and of the objects being manipulated, and the dynamics level updates the objects’ states by simulating gravity and the forces between them. (b) Given a reward, a specified final configuration, GenEA plans with a task planner, a motion planner and a forward engine.

From Figure 3: Reasoning queries in GenEA. (a) An action inference trial: the model gets the initial and final block ... — From Figure 3: Reasoning queries in GenEA. (a) An action inference trial: the model gets the initial and final block configurations and two candidate first actions, a one-handed pick of the red block and a two-handed pick of the blue and yellow blocks. (b) The distribution of plans that reach the final configuration; textured bars mark plans that also satisfy each candidate action. (c) An efficient plan generated by GenEA that takes advantage of gravity, made of two one-handed actions.

From Figure 4: Explaining social inferences. (a, b) Example stimuli from the goal inference and action inference tria... — From Figure 4: Explaining social inferences. (a, b) Example stimuli from the goal inference and action inference trials of Experiment 1. (c) GenEA strongly correlates with human ratings. (d) GenEA achieves a significantly stronger correlation with participant judgments than the alternatives, including TAMP and heuristic models. (e) Example stimuli from Experiment 2, where participants report whether the agent who reconfigured the blocks used one hand or two. (f, g) GenEA closely captures human intuitions in this task and again correlates more strongly than the alternatives.

From Figure 6: Testing GenEA in navigation. (a) Example stimuli from Experiment 3: an agent stands on the left-most s... — From Figure 6: Testing GenEA in navigation. (a) Example stimuli from Experiment 3: an agent stands on the left-most start platform, with one or more intermediate platforms and the final platform on the right; participants judge how difficult it would be to reach the right-most platform. (b) GenEA closely captures people’s difficulty judgments (r = .91). (c) GenEA explains more variance than the alternatives, including TAMP and a stability heuristic. (d) GenEA and humans, but not TAMP, judge trials with wooden platforms to be on average more difficult than those with stone platforms.

Abstract

Reasoning about people's plans and goals based on their actions is fundamental to human social interaction and essential for cooperative, embodied artificial intelligence. Formalizing this capacity in naturalistic, embodied settings has been a longstanding challenge. Here, we present GenEA, a novel agent representation capable of reasoning over multi-step, long-horizon embodied plans. At the heart of GenEA lies a multilevel state-space that couples the coarse biomechanics of an articulated body and its interactions with nearby objects by extending the task-and-motion planning framework from robotics with simulation-based intuitive physics. By embedding this state-space in a generative model, GenEA yields a joint probability distribution over goals and embodied plans. Across several experiments spanning the qualitatively distinct domains of object manipulation and navigation, featuring diverse tasks and multiple behavioral measures, we show that GenEA, through simple transformations of this joint distribution, recapitulates how humans reason about embodied plans. Crucially, GenEA consistently outperforms strong alternative models, including plausible heuristics, standard task-and-motion planning, and a state-of-the-art vision-language model. GenEA thereby offers a new perspective on the architecture of human reasoning and social cognition, suggesting that embodied artificial intelligence systems seeking to seamlessly interact with humans should adopt GenEA-like generative models of action planning.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong