Thermo-VLA: A Thermal-Aware Vision-Language-Action Model with Gated Residual Modulation for Robotic Manipulation

Jingxuan Chen, Junshi Luo, Chencheng Dong, Xiaojian Nie, Tongge Huang, Qiang Huang

November 2026

Real robot, colour-conditioned Task VIII: picking the instructed cup out of six and placing it on the black coaster.

TL;DR Thermo-VLA splits a thermal image into cold and hot views and adds them to the RGB tokens of pi_0.5 as language-gated residuals. On a dual-arm Unitree G1 it averages 0.86 success on four temperature-dependent tasks, 18 points above pi_0.5 given the thermal image as an extra camera.

What they do

VLAs that take thermal images usually treat them as one more camera, which helps little when an instruction depends on temperature (“the coldest cup”, “the one with no heat signature”). Thermo-VLA adds a ThermoRes Encoder to pi_0.5 that turns the thermal image into cold and hot attribute tokens and injects them into the head-camera RGB tokens. It is evaluated on six real tasks on a dual-arm Unitree G1, trained on 1,307 demonstrations.

How it works

Cold/hot decomposition. The thermal image is calibrated to a background reference estimated from patches near the top of the frame, then split into two views that emphasise responses below and above the background. A shared SigLIP-based encoder and the RGB projector turn both into tokens with the RGB token layout.
Gated residual injection. Text tokens cross-attend to the cold and hot tokens, and a small gate MLP predicts one weight per branch. The weighted thermal tokens are added to the RGB tokens as residuals, so the RGB path stays an identity path.
Action generation. The modulated head tokens, wrist-camera tokens and text form the prefix for the Gemma backbone and the flow-matching action expert, unchanged from the base model.

Results

Four temperature-dependent tasks, 20 trials per instruction: average success 0.86, 18 points above pi_0.5 with RGB plus thermal input. On the task of finding the target with an abnormal thermal response it succeeds in every trial, while the other RGB-thermal methods are around 0.5.
Generalisation: 0.79 on unseen objects (0.64 for pi_0.5 RGBT, 0.38 for ThermoAct), 0.76 under red illumination and 0.79 on unseen tablecloths.
Forcing the thermal residual to zero drops temperature-dependent tasks to about 0.5: the robot still reaches and grasps but picks targets at chance.
The same residual design on colour channels raises pi_0.5 from 0.572 to 0.775 average success on colour-conditioned tasks in LIBERO and on the real robot.

Limitations

On the two RGB-only tasks Thermo-VLA is slightly below plain pi_0.5; the authors plan to pretrain the thermal encoder on more thermal data. Relative temperature ranking stays hard (0.58 on the coffee task), since the cold/hot split carries no temperature magnitude, and reassigning temperatures to seen objects drops success to 0.60.

Videos

The same cup task on a red-blue patterned tablecloth.

Simulation, Task VII: a Franka places the instructed coloured cube on a plate over a patterned tablecloth.

Figures

From Figure 2: Overview of the Thermo-VLA framework. RGB observations are encoded by the RGB visual encoder, while la... — From Figure 2: Overview of the Thermo-VLA framework. RGB observations are encoded by the RGB visual encoder, while language instructions and robot states are mapped to textual embeddings. ThermoRes Encoder decomposes thermal observations into cold and hot attributes and injects them into RGB tokens through context-gated residuals. The resulting multimodal prefix is used for flow-matching policy training and continuous action generation.

From Figure 3: Representative successful and failed executions. Left: successful executions of Thermo-VLA. Right: rep... — From Figure 3: Representative successful and failed executions. Left: successful executions of Thermo-VLA. Right: representative failure cases of competing RGB–thermal methods. The bottom row shows the corresponding attention maps.

From Figure 4: Generalisation settings and representative successful executions of Thermo-VLA. The first two columns ... — From Figure 4: Generalisation settings and representative successful executions of Thermo-VLA. The first two columns show the original and modified scenes, with only the corresponding generalisation factor changed; the remaining columns show successful executions. From top to bottom: unseen bottle, unseen objects, temperature reassignment, illumination change, and unseen tablecloth.

From Figure 5: RGBRes Encoder and representative scenes from the additional colour-conditioned tasks. Task VII is sho... — From Figure 5: RGBRes Encoder and representative scenes from the additional colour-conditioned tasks. Task VII is shown in simulation and Task VIII in the real world.

Abstract

Recent Vision-Language-Action (VLA) models have begun incorporating thermal sensing, but existing approaches typically treat thermal images as ordinary visual inputs, providing limited support for temperature-aware relational reasoning in manipulation. To address this limitation, we propose Thermo-VLA with a ThermoRes Encoder, which decomposes thermal images into cold and hot attribute representations and injects them into the RGB token through language-conditioned residual modulation, enabling temperature-driven target selection and continuous manipulation. We further construct six real-world robotic tasks on a dual-arm Unitree G1 platform, spanning basic grounding to relational reasoning. Experiments show that Thermo-VLA achieves an average success rate of 0.86 across four thermal-dependent tasks, outperforming a pi_0.5 baseline that directly incorporates RGB-thermal observations by 18 percentage points. Under unseen-object and scene variations, Thermo-VLA maintains an average success rate of 0.78, achieving the best generalisation performance among the compared methods. Beyond thermal sensing, we transfer the same residual modulation mechanism to colour attributes and observe consistent improvements in both simulation and real-robot experiments, indicating its potential to generalise beyond thermal attributes. Project website: https://j-x-chen.github.io/Thermo-VLA-web/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong