HeLM: Hierarchical Explicit Language Memory for Non-Markovian Manipulation

Gyeonghun Kim, Donghoon Kim, Minsoo Choi, Suyun Lee

November 2026

Real-time run of Press in Order (A2), with the language memory shown next to the robot view.

TL;DR A fine-tuned VLM keeps a running text memory of task events, such as which buttons were already pressed, and feeds commands to a π0 low-level policy, so the robot can count, follow orders and recall information across instructions. On 10 real-robot memory tasks it reaches 85.7% subtask success within episodes (vs 72.9% for the best baseline) and 89.2% across episodes.

What they do

Many manipulation tasks are non-Markovian: after pressing a button twice or wiping part of a window, the scene can look just as it did before, so a policy that sees only the current frame loses track. HeLM gives a VLA an explicit text memory. A fine-tuned VLM (the high-level policy) records progress in language and issues commands to a π0 low-level policy. The memory also carries over between instructions, which the authors test with five new inter-episode tasks.

How it works

Two memory fields. Working memory tracks progress within an episode (e.g. “Progress: blue (Goal: blue, green, red)”). When the instruction changes it is summarized into an episodic context that persists into the next episode.
Two-stage inference. Every 10 low-level steps, Qwen2.5-VL-7B checks the frame for an event from a predefined list. Only if it finds one, or the instruction changes, does a second prompt rewrite the memory.
TaskSpec data. A per-task schema of commands, events and memory transitions turns short annotated subtask episodes into about 130k frame-plus-memory training samples.

Results

Real robot: a 7-DoF AgileX PiPER; every method uses π0 fine-tuned on 150-200 demos per task.

Intra-episode tasks: 85.7% subtask success vs 72.9% for Keyframe-HLP (the same hierarchy with image keyframes as memory), 39.5% for PCMB-π0 and 16.7% for plain π0. Strict success is 81.3% vs 29.3%, 0% and 0%.
Inter-episode tasks: 89.2% vs 16.6% for Keyframe-HLP. Held-out compositions of seen subtasks: 78% vs 65.7%.
A one-stage variant that updates memory on every call drops to 67.7%. At long horizons HeLM runs 2.5x faster than Keyframe-HLP and stays under 6 GB of VRAM, against 28 GB for an 8-event episode.

Limitations

The events worth remembering must be predefined per task, and only discrete events are covered, not continuous state such as liquid levels. Most failures are false-positive detections (e.g. a drawer judged open when it is not), which the memory cannot self-correct, or missed presses by the low-level policy. Synchronous calls at chunk boundaries add latency and can miss events that happen within a chunk.

Videos

The same run later on, switching between detect mode and memory updates after each press.

Figures

From Figure 1: A non-Markovian task. A vanilla VLA loses track after pressing the red button and presses it again; He... — From Figure 1: A non-Markovian task. A vanilla VLA loses track after pressing the red button and presses it again; HeLM stores progress in language memory, so the high-level policy guides the low-level policy correctly.

From Figure 2: HeLM architecture. The high-level policy, a pretrained VLM with language memory, switches between a de... — From Figure 2: HeLM architecture. The high-level policy, a pretrained VLM with language memory, switches between a detect mode that watches for events and an update mode that writes them into memory. The low-level policy, a pretrained VLA, executes the action commands it generates.

From Figure 4: The real-world task suite. Intra-episode tasks (A1-A5, top) test working memory within a single instru... — From Figure 4: The real-world task suite. Intra-episode tasks (A1-A5, top) test working memory within a single instruction; inter-episode tasks (E1-E5, bottom) test context carried across task boundaries.

From Figure 5: Intra-episode results. HeLM has the highest average subtask success rate; Keyframe-HLP fails to termin... — From Figure 5: Intra-episode results. HeLM has the highest average subtask success rate; Keyframe-HLP fails to terminate on A1 and A4, and PCMB-π0 and Naive-π0 fail both sequencing and termination in all trials. Bars are means over 15-30 trials per task, error bars one standard deviation across variants.

Abstract

Recent advances in Vision-Language-Action (VLA) models have enabled fine manipulation capabilities, yet performing tasks that require working memory remains a key challenge. Most existing VLA architectures operate under a Markovian assumption, mapping only the current observation to an action, which leads to failure in non-Markovian environments. Inspired by the human 'inner monologue', we propose Hierarchical Explicit Language Memory (HeLM), a hierarchical VLA framework that maintains task-relevant history in explicit language through factorized event detection and event-triggered memory updates. We evaluate HeLM across 10 non-Markovian real-world tasks with 56 variants, including five newly formulated inter-episode tasks that require persistent context across task boundaries. HeLM improves intra-episode success by 12.8 percentage points over the strongest baseline and achieves 78% subtask success on compositional generalization tasks, while using up to 4.5x less memory and running 2.5x faster than the keyframe baseline at long horizons.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong