Control-Aware Command Generation for Asynchronous Control and Action Inference with Vision-Based Policies towards Fast Execution

Takayuki Osa, Kazuki Ota

November 2026

TL;DR A tracking-control layer turns the jumpy action chunks of an asynchronously running policy into smooth, acceleration-limited joint targets, and skips actions when successive predictions agree. On a bimanual ALOHA with ACT, tasks take 53-63% of the synchronous execution time and pen-storage success rises from 30% to 83%.

What they do

Vision-based policies such as ACT and OpenVLA are slow to execute because the robot waits for each action inference. Running inference asynchronously removes the wait, but new action chunks arrive late and do not line up with the old ones, so the commands jump. This paper fixes the problem at the control level, without retraining the policy: a command generator tracks the policy’s outputs smoothly, and an adaptive rule skips actions when the predictions are reliable.

How it works

Control-aware command generation (CAC). Policy outputs are treated as desired joint positions. The command sent to the robot follows them with a feedforward term plus a saturated feedback term, analogous to sliding mode control, and its acceleration is clipped to fixed bounds.
Two threads. A control thread sends commands and picks up a new action chunk whenever the inference thread has one; otherwise it keeps using the previous chunk. Observations and chunks carry time stamps.
Adaptive skipping. Several chunks predict the action for the same step. If the variance of those predictions is below a threshold (0.2 in all experiments), the step is skipped and time advances by two steps.

Results

Real bimanual ALOHA, three tasks, 50 demonstrations each, 10 trials for each of three seeds.
ACT with CAC, asynchronous inference and adaptive skipping takes 53-63% of the time of synchronous ACT: box handover 8.6 s vs 16.2 s, case stacking 19.2 s vs 34.4 s, pen storage 34.4 s vs 52.6 s.
Pen storage success rises from 30.0% (synchronous ACT) to 83.3%; the other two tasks stay at 96.7-100%. Uniform skipping instead drops pen storage to 0%.
With OpenVLA-OFT on pen storage, success stays at 8/10 while episode duration falls from 68.2 s to 59.0 s; without CAC, asynchronous OpenVLA-OFT oscillated and could not reliably complete the task.

Limitations

There is no comparison with generic smoothing methods or with Real-Time Chunking, the inpainting approach for diffusion and flow-matching policies. OpenVLA-OFT was run with default parameters, including chunk size, on one task with 10 trials. Evaluation covers one robot and three tasks.

Videos

Figures

From Figure 1: The robot control thread and the action inference thread run asynchronously, sharing observations and ... — From Figure 1: The robot control thread and the action inference thread run asynchronously, sharing observations and action chunks. The control thread turns the inferred actions into smooth commands.

From Figure 4: Adaptive action skipping. Where overlapping action chunks agree (low variance) the robot moves fast; w... — From Figure 4: Adaptive action skipping. Where overlapping action chunks agree (low variance) the robot moves fast; where they disagree (high variance) it moves slowly.

From Figure 5: The pen storage task on the bimanual robot: open the drawer, hand over the pen, put the pen in the dra... — From Figure 5: The pen storage task on the bimanual robot: open the drawer, hand over the pen, put the pen in the drawer, close the drawer.

From Figure 6: Distribution of the jerk of the commands for the five ACT variants. The variants with control-aware co... — From Figure 6: Distribution of the jerk of the commands for the five ACT variants. The variants with control-aware command generation (CAC) have much lower jerk.

Abstract

Vision-based control policies, including vision-language-action models, have enabled robots to perform increasingly complex tasks. However, their execution speed is often limited by the high computational cost of action inference. In this work, we investigate what can be achieved at the control level without modifying the underlying policy learning process. We introduce an asynchronous inference and control framework that enables fast task execution while maintaining smooth and stable robot motions. By formulating the generation of continuous target positions from discontinuous model predictions as a tracking control problem, our method makes asynchronous inference practical for real robots. We further accelerate execution through adaptive action skipping based on the variance of predicted positions. Experiments with Action Chunking Transformer and OpenVLA-OFT on a bimanual robot demonstrate substantial speedups while preserving high task success rates.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong