SomaAlign: Reliability-Aware Multimodal Representation Learning for Soft Robotic Grasping

Haoyuan Wang, Shuai Peng, Yongkai Liao, Yuxiao Sun, Wei Luo, Xinxing Chen, Jian Huang

November 2026

TL;DR A soft-gripper grasping dataset with six synchronized modalities (two RGB views, depth, tactile pressure, IMU and text) over 100 objects, plus a contrastive method that sets each modality pair's alignment strength by how reliably that pair can be reconstructed. It reaches 90.66% average Recall@10 across 30 cross-modal retrieval directions, and its frozen encoders classify grasp outcome on a separately collected 998-trial set with 94.59% accuracy.

What they do

Soft grippers deform, vibrate and slip, so their tactile and IMU signals are noisier than camera images, and most visuo-tactile datasets use rigid grippers with optical tactile sensors. The authors collect Soma, a soft-gripper grasping dataset with six synchronized modalities, and Soma-Success, a separate grasp-outcome set. They then propose SomaAlign, which embeds all six modalities in one space while accounting for how reliable each modality is.

How it works

Data. Pneumatic two- and three-finger grippers carry a hand camera, one IMU per finger and 8×14 pressure arrays on the finger pads, next to a remote RealSense D435i. A motorized rod pulls each grasped object down on a fishing line until it slips or drops. Ten trajectories per object give 74,914 samples labelled stationary or sliding, with templated text.
Stage I. Masked reconstruction per modality; modalities that are harder to reconstruct get a lower reliability score.
Stage II. Pairwise InfoNCE on UniTouch-initialized encoders, with a temperature per modality pair set by its reliability, so reliable pairs are aligned more sharply.
Stage III. A fuzzy score of cross-modal conflict, averaged over the batch, scales how strongly each sample’s modalities are pulled together, while a penalty keeps sample centers near their Stage-II positions.

Results

Retrieval over 30 modality directions: R@1 63.04%, against 44.55% for ImageBind-style hub alignment and 42.00% for pairwise InfoNCE with the same encoders.
Vision-tactile-language retrieval: average R@1 69.71% vs 55.64% for adapted TVL, the strongest published baseline.
Soma-Success with frozen encoders and object-grouped 5-fold probes: 94.59% post-hoc outcome accuracy (MLP). Early prediction before lifting reaches 84.07%, below hub alignment’s 86.48%.
Removing Stage III drops R@1 to 22.72%.

Limitations

The reliability score is a reconstruction proxy, not a measure of sensor noise. Text annotations are templated, so language is limited. Grasp-outcome results use trained probes rather than zero-shot transfer, and other platforms, sensors and end-effectors are untested. The representations are not yet used for closed-loop control.

Figures

From Figure 1: The soft robotic platform records six modalities: remote RGB, hand RGB, depth, tactile, IMU and text. ... — From Figure 1: The soft robotic platform records six modalities: remote RGB, hand RGB, depth, tactile, IMU and text. Soma has 100 objects, 1,000 trajectories and 74,914 aligned samples; Soma-Success has 998 valid grasp trials. SomaAlign learns a shared representation through reliability assessment, adaptive feature alignment and conflict-aware refinement, used for retrieval, attribute recognition, question answering, cross-modal generation and grasp success prediction.

From Figure 2: The Soma acquisition system. One software polling loop reads the remote RGB-D camera, the hand-mounted... — From Figure 2: The Soma acquisition system. One software polling loop reads the remote RGB-D camera, the hand-mounted camera, finger-mounted IMUs and flexible tactile pressure arrays. Text descriptions are generated from experiment metadata with a template. Sample sensor outputs and a text annotation are shown on the right.

From Figure 5: The three SomaAlign stages. (1) Masked-reconstruction difficulty gives each modality a reliability sco... — From Figure 5: The three SomaAlign stages. (1) Masked-reconstruction difficulty gives each modality a reliability score, combined into pairwise scores. (2) Contrastive alignment uses those scores to set each modality pair's temperature. (3) Conflict-aware refinement estimates sample-level cross-modal conflict, scales how strongly each sample's modalities are pulled together, and anchors sample centers to the frozen Stage-II geometry. Only the modality encoders are kept for inference.

From Figure 4: The Soma-Success collection protocol. Three snapshots are taken at the pre-grasp, closed-gripper and e... — From Figure 4: The Soma-Success collection protocol. Three snapshots are taken at the pre-grasp, closed-gripper and end-of-lift stages, with the same rule for successful and failed trials. Examples are shown for the two-finger and three-finger grippers.

Abstract

Robust multimodal perception for soft robotic grasping is challenged by modality-dependent noise, semantic ambiguity, and limited coordinated multisensory data. We introduce Soma, a dataset containing 100 objects, 1,000 continuous grasping trajectories, and 74,914 multimodal samples grouped by software acquisition cycle, spanning remote RGB, hand RGB, depth, tactile, IMU, and structured text observations. We also collect Soma-Success, an independent grasp-outcome dataset with 1,000 trials (998 retained after quality control), to evaluate frozen-encoder transfer for post-hoc grasp outcome classification and early grasp success prediction. Built on Soma, SomaAlign learns multimodal representations in three stages: masked reconstruction estimates relative modality reliability; reliability-aware contrastive alignment uses pair-specific adaptive temperatures to adjust alignment sharpness; and conflict-aware semantic constraints improve intra-sample cross-modal consistency while preserving global feature geometry. SomaAlign achieves an average Recall@10 of 90.66% across 30 directed cross-modal retrieval tasks. The learned representations also support physical attribute recognition, multimodal question answering, cross-modal generation, and cross-dataset transfer to grasp outcome prediction. On Soma-Success, frozen SomaAlign encoders with lightweight probes achieve 94.59% post-hoc classification accuracy and 84.07% early prediction accuracy. Together, Soma, Soma-Success, and SomaAlign provide a data and representation-learning foundation for robust and transferable multimodal perception in soft robotic manipulation.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong