Physics-Informed Tactile-based Shape Estimation of Deformable Linear Objects

TL;DR Four DIGIT tactile sensors on two grippers, plus the arm's pose, let a ViT reconstruct the full 3D shape of a wire held between them with no camera. On a 550 mm steel wire it reaches 30.5 mm RMSE, against 55.0 mm from gripper pose alone.
What they do
Cameras lose sight of cables when grippers, fixtures or clutter block the view. This letter asks whether the full 3D shape of a deformable linear object (DLO) held at both ends can be recovered from touch and proprioception alone. Two Robotiq 2F-85 grippers hold the ends of a wire. One is fixed, the other rides on a Kinova Gen3 arm, and every finger pad is a DIGIT tactile sensor. The wire can leave each grasp at any angle, so the tactile images also have to tell the model which way it points. Training labels come from an OptiTrack system tracking 14 markers on the wire, which is used only for training: 196,678 samples over 36 episodes on a 1.5 mm, 550 mm braided steel wire.
How it works
- Tactile encoder. For each sensor, the current and previous frames (minus a no-contact reference image) go into a ViT-S. The ViT-S starts from the Sparsh tactile foundation model and is mostly frozen. Features from layers 3, 6, 9 and 12 are mixed with learned weights.
- Pose queries. An MLP turns the moving gripper’s 7-D pose into one query token per B-spline control point. A cross-attention layer lets these queries read the tactile tokens, then a self-attention layer passes information along the wire.
- Smooth output. The predicted control points go through a parameter-free, differentiable cubic B-spline layer.
- Physics-informed loss. Besides vertex positions, the loss matches tangents, edge lengths and discrete curvature taken from the Discrete Elastic Rods formulation, plus a Chamfer term.
Results
- On the steel-wire test set (21,853 samples), RMSE is 30.5 mm and Chamfer distance 27.3 mm. With gripper pose alone they are 55.0 and 44.0 mm. Pose plus the tip direction extracted from the tactile images gives an RMSE of 50.6 mm, and a ResNet-18 tactile encoder gives 36.0 mm. Removing the physics loss or the B-spline layer raises RMSE to 37.0 or 35.0 mm. The model runs at 16 Hz.
- Reliable estimates need only 15 of the 36 episodes (about 42% of the data).
- Finetuning on a USB cable and on a 2 mm, 750 mm steel wire beats training from scratch on the same 50,000 samples. Without finetuning, the model does poorly on the new cables.
- With 2 of the 4 sensors removed, a retrained model reaches 37.8 mm RMSE. Blanking 2 sensors at test time without retraining gives 70.9 mm.
Limitations
The setup is quasi-static, with both ends grasped, grasp angles within about ±45°, and motion-capture labels needed for every new cable. Without retraining, the model does not handle a different grasped length: at 500 mm or 580 mm, mean error rises to 36 or 40 mm. Accuracy falls sharply when sensors fail suddenly. The authors name near-buckling and rapid slippage as states outside the training distribution.
Figures




Abstract
Accurate shape estimation of Deformable Linear Objects (DLOs) is a fundamental challenge in robotic manipulation, often hindered by visual occlusions and the high-dimensional configuration space of flexible materials. While Force/Torque (F/T) and vision-centric methods have been proposed, they frequently struggle with grasp-site ambiguities and environmental interference. In this letter, we investigate the hypothesis that global shape estimation can be achieved through tactile sensing and proprioception, without any visual perception. We contend that dense, high-resolution tactile imprints capture critical boundary conditions and contact gradients that are sufficient to resolve the high-dimensional configuration of the DLO. To prove this, we introduce a Physics-Informed Vision Transformer (ViT) framework that maps tactile images to a 3D vertex chain. By leveraging self-attention mechanisms and a physics-weighted loss function, our model enforces structural priors to ensure that the predicted geometry adheres to the object's physical manifold. Experimental results demonstrate that our tactile-based approach achieves superior reconstruction accuracy across Euclidean and Chamfer metrics. Furthermore, we demonstrate the efficiency of this modality, achieving model transfer to novel DLO materials and geometries. Our findings suggest that tactile feedback provides a robust and computationally efficient alternative for DLO state estimation in visually degraded or occluded environments.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
