Tactile sensing
Sensors that let robots feel contact, and policies that use touch.
Start here: seminal papers
Deep Learning for Tactile Understanding From Visual and Haptic DataSeminal
November 2015
CNNs label household objects with 24 haptic adjectives such as compressible or smooth, from BioTac touch signals, from images, or from both. On PHAC-2 the haptic CNN reaches 83.2 AUC against 78.2 for hand-designed features, and combining touch and vision reaches 85.9.
Shape-independent Hardness Estimation Using Deep Learning and a GelSight Tactile SensorSeminal
April 2017
A CNN+LSTM regresses object hardness directly from GelSight tactile videos of loosely controlled presses by a human or a robot gripper, without knowing the object's shape. On trained shapes with unseen hardness values it reaches R² 0.96 (RMSE 5.18 Shore 00); on novel shapes R² drops to 0.79.
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?Seminal
October 2017
A network predicts whether a grasp will hold from a camera image plus two GelSight fingertip images, trained on 9,269 self-collected grasps. Adding touch lifts real-robot grasp success on 12 unseen objects from 80% (vision only) to 94%.
GelSlim: A High-Resolution, Compact, Robust, and Calibrated Tactile-sensing Finger★ Must read
March 2018
A GelSight-style tactile finger made slim for grasping in clutter by routing light through acrylic guides and imaging the gel via a mirror, with a fabric-covered gel and an image calibration that offsets wear. After 3000 grasps fewer than 0.06% of its pixels had died.
Towards Robotic Feeding: Role of Haptics in Fork-based Food ManipulationSeminal
April 2018
They recorded 3304 trials of 12 people feeding a mannequin with a force/torque-sensing fork, built a taxonomy of feeding strategies, and classified food into four compliance categories from haptic and motion signals, with a TCN reaching 80.47% accuracy. A robot using one fixed vertical skewering motion acquired bites less reliably than people.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch★ Must read
May 2018
A network that sees a camera image and two GelSight fingertip images predicts whether a candidate grasp adjustment will succeed, and the robot regrasps by picking the best one. On 11 unseen objects it grasps 94% of the time, against 76% for a hand-tuned baseline.
Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks★ Must read
October 2018
A self-supervised network fuses camera images, force-torque readings and proprioception into a compact representation, and a peg-insertion policy is learned on top of it with RL on a real robot. In simulation the full model completes nearly 80% of insertions, while every baseline missing vision or touch completes under 5%.
Manipulation by Feel: Touch-Based Control with Deep Predictive ModelsSeminal
March 2019
A video-prediction model learned from random pokes predicts future GelSight tactile images, and MPC uses it to push a ball, analog stick or 20-sided die until the sensor reading matches a goal tactile image. On die rolling it reaches the target face in 26 of 30 trials, against 14 of 30 for a hand-designed centroid controller.
Connecting Touch and Vision via Cross-Modal PredictionSeminal
June 2019
A KUKA arm with a GelSight sensor recorded 12,000 touches on 195 objects, and a conditional GAN given reference images and nearby frames learns to predict touch from vision and vision from touch. Raters judged its predicted touch locations to feel like the true ones 89% of the time on seen objects, close to a hand-supervised baseline at 90%.
Cable Manipulation with a Tactile-Reactive GripperSeminal
October 2019
A gripper with GelSight fingertips follows a free-hanging cable by sliding along it, using only touch to control grip force and cable pose. With an LQR pose controller it follows 98.2% of the length of five different cables, and it inserts a headphone plug in 85% of 20 trials.
OmniTact: A Multi-Directional High Resolution Touch SensorSeminal
March 2020
OmniTact casts a GelSight-style gel skin directly over five micro-cameras to make a thumb-sized fingertip with high-resolution touch on all sides. With both its top and side cameras, a learned policy inserts an electrical connector in 80% of trials, against 17% with an OptoForce sensor.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation★ Must read
May 2020
The open-source, fingertip-sized GelSight-style sensor that made camera-based touch cheap and repeatable enough to put on multi-fingered hands. Shown driving learned tactile model-predictive control of glass marbles in hand.
3D Shape Reconstruction from Vision and TouchSeminal
July 2020
Reconstructs an object's 3D mesh from one RGB image plus four simulated high-resolution touch readings from a grasping robot hand, by deforming mesh patches from vision around exact patches from touch. Adding touch lowers test Chamfer distance from 1.074 to 0.991 with the hand occluding the view, and five grasps beat one (0.654 vs 0.804).
TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile SensorsSeminal
December 2020
TACTO is an open-source simulator that renders camera images from vision-based tactile sensors like DIGIT and OmniTact by syncing object poses from PyBullet into an OpenGL scene, reaching 200 frames per second for one DIGIT on GPU. The authors use it to collect 1 million simulated grasps for grasp-stability learning and to learn a marble-rolling controller in 8 minutes.
Generation of GelSight Tactile Images for Sim2Real LearningSeminal
January 2021
They simulate a GelSight optical tactile sensor in Gazebo by turning a depth-camera image into a smoothed elastomer heightmap and shading it with Phong's model. A classifier trained only on generated images, with texture augmentation, recognises 21 real objects with 76.19% test accuracy.
GelSlim3.0: High-Resolution Measurement of Shape, Force and Slip in a Compact Tactile-Sensing FingerSeminal
March 2021
GelSlim 3.0 is a 20 mm-thick, open-source GelSight-style finger whose LEDs and light-shaping lens are optimized in optical simulation so photometric stereo works on a flat gel, giving depth, slip and dense 3D force from one camera. It weighs 45 g, costs about $25 in parts and has a 675 mm² sensing field, twice that of Digit at a comparable thickness.
Sim-to-Real for Robotic Tactile Sensing via Physics-Based Simulation and Learned Latent ProjectionsSeminal
March 2021
A finite-element model of the BioTac fingertip in GPU-based Isaac Gym matches a validated ANSYS model at 75x the speed. Small networks that project between learned latent spaces of simulated skin deformation and real electrode signals then predict real BioTac output for unseen objects and recover contact patches from real signals.
Tactile-RL for Insertion: Generalization to Objects of Unknown GeometrySeminal
April 2021
An RL policy learns on a real robot to correct peg-in-hole misalignment from two GelSlim tactile sensors, using a curriculum of increasingly constrained environments and marker-flow input. Trained on 4 objects, it inserts 4 unseen everyday objects with over 85% average success.
Soft Biomimetic Optical Tactile Sensing with the TacTip: A ReviewSeminal
May 2021
A review of the TacTip, a soft optical tactile sensor whose internal pins mimic human skin and lever surface indentation into marker motion seen by an internal camera. It covers a decade of TacTip sensors, hands and skills, with localization accuracy improving from about 1 mm to 0.1-0.5 mm once ConvNets were applied to the tactile images.
Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image TranslationSeminal
June 2021
Train tactile RL policies in PyBullet on simulated depth images of a TacTip sensor, then run them on a real robot by translating real tactile images into simulated ones with a pix2pix GAN. Zero-shot transfer works on 4 of 5 tasks, including 100 consecutive successful ball-rolling trials.
GelSight Wedge: Measuring High-Resolution 3D Contact Geometry with a Compact Robot FingerSeminal
June 2021
GelSight Wedge is a compact tactile robot finger that keeps GelSight's high-resolution 3D reconstruction, with lights on only three sides so the tip can pick up small objects. When a light direction is missing, a U-Net trained on synthetic surfaces fills in the gradient and cuts the angle error from 5.7° to 2.7°.
Taxim: An Example-based Simulation Model for GelSight Tactile SensorsSeminal
September 2021
A GelSight simulator calibrated from fewer than 100 real presses. A polynomial lookup table renders the tactile images and FEM-calibrated superposition moves the markers. Its images match real ones better than three physics-based simulators (MSE 58.4 vs 90.6-215.9), and it runs at 18.1 fps on a CPU.
ReSkin: versatile, replaceable, lasting tactile skinsSeminal
October 2021
A 2-3 mm magnetic skin you can peel off and replace: magnetized particles in elastomer over a few magnetometers, with learned (and self-supervised) models that stay accurate across replacement skins and 50k+ contacts.
DenseTact: Optical Tactile Sensor for Dense Shape ReconstructionSeminal
January 2022
A hemispherical camera-based fingertip that reconstructs the 3D shape of its whole contact surface from one fisheye image using a DenseNet encoder-decoder. Mean depth error on unseen shapes is 0.28 mm at 18 ms per image, for under $80 per sensor.
MidasTouch: Monte-Carlo inference over distributions across sliding touchSeminal
October 2022
A particle filter localizes a DIGIT touch sensor sliding over a known object by matching learned codes of local 3D tactile geometry against a precomputed per-object codebook. Over 500 real trials on 10 YCB objects, the final pose error averages 1.97 cm and 21.48°.
Touch and Go: Learning from Human-Collected Vision and TouchSeminal
November 2022
People walked through indoor and outdoor scenes pressing a GelSight sensor onto objects while filming, collecting 13.9k touches of about 3971 objects. Tactile features learned from these image-touch pairs beat ImageNet features at material recognition (54.7% vs 46.9%) and robot grasp-outcome prediction (78.1% vs 73.0%).
See, Hear, and Feel: Smart Sensory Fusion for Robotic ManipulationSeminal
December 2022
A robot arm with a camera, a contact microphone and GelSight fingertip sensors fuses all three senses with self-attention and learns dense packing and pouring from demonstrations. With all three it inserts a peg past hidden bumps in 100% of trials (vision alone 25%) and pours to within 1.41 g of a 40 g target.
Rotating without Seeing: Towards In-hand Dexterity through TouchSeminal
March 2023
Covers one side of an Allegro hand with 16 cheap binary touch sensors and trains an RL policy in simulation that rotates objects in the palm from touch and joint angles alone. On the real hand it turns a cuboid 4.91 rounds in 30 s versus 0.25 without touch, and rotates unseen objects such as an orange and a soup can.
9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force EstimationSeminal
August 2023
A $15 open-source tactile sensor, 32.5 x 25.5 x 25.5 mm, that reconstructs contact shape and estimates 6D force from the raw gel image, with no markers. A DenseNet trained on 100,417 image-force pairs from 175 objects reaches 0.370 N and 0.0077 Nm mean error on unseen objects.
General In-Hand Object Rotation with Vision and TouchSeminal
September 2023
RotateIt trains an Allegro hand in simulation to rotate objects with its fingertips about the x, y or z axis, using a transformer that fuses segmented depth, fingertip contact and proprioception. Vision and touch lift x-axis rotation reward from 79 to 118 (privileged oracle 125), and the policy transfers to real objects without fine-tuning.
Robot Synesthesia: In-Hand Manipulation with Visuotactile SensingSeminal
December 2023
Turn each triggered touch sensor into points on its mesh and merge them with the depth-camera point cloud, so one PointNet policy sees vision and touch in the same 3D space. Trained in simulation and transferred zero-shot to an Allegro hand, it reaches 22.9 rounds of real double-ball rotation, against 18.8 when touch is fed in as a separate vector.
Neural feels with neural fields: Visuo-tactile perception for in-hand manipulationSeminal
December 2023
NeuralFeels fuses a camera, DIGIT touch sensors on four fingertips and joint angles to learn an object's neural SDF online and track its pose while an Allegro hand rotates it. Across 70 simulated and real trials it reaches an 81% reconstruction F-score with 4.7 mm average pose drift (2.3 mm when the CAD model is given).
Binding Touch to Everything: Learning Unified Multimodal Tactile RepresentationsSeminal
January 2024
UniTouch trains one touch encoder for several vision-based tactile sensors by aligning it to a frozen ImageBind image embedding, which links touch to text and audio without paired data. Zero-shot, it beats supervised methods at touch-to-vision retrieval on ObjectFolder 2.0 (41.9 vs 32.3 mAP).
DIFFTACTILE: A Physics-based Differentiable Tactile Simulator for Contact-rich Robotic ManipulationSeminal
March 2024
A differentiable tactile simulator that couples an FEM model of a GelSight elastomer to MPM and PBD objects (rigid, elastic, elastoplastic, articulated, cable) through a penalty contact model. Gradient-based trajectory optimization with its tactile losses turns a block 60.8 degrees against a wall, where CMA-ES, PPO and SAC reach at most 5 degrees.
Learning Visuotactile Skills with Two Multifingered HandsSeminal
April 2024
HATO is a low-cost VR teleoperation system for two UR5e arms fitted with touch-sensing prosthetic hands, used to train diffusion policies on vision, touch and proprioception. The policies succeed 9-10/10 on handover, stacking and pouring and 5/10 on steak serving, which drops to 0/10 without touch.
TacSL: A Library for Visuotactile Sensor Simulation and LearningSeminal
August 2024
TacSL simulates GelSight tactile images and contact force fields on the GPU inside Isaac Simulator, over 200x faster than prior tactile simulators, and adds a policy-learning toolkit with a new RL method (AACD). Tactile policies trained in simulation transfer zero-shot to a real robot, succeeding 91.4% of the time on peg placement and 82.7% on peg insertion.
Sparsh: Self-supervised touch representations for vision-based tactile sensingSeminal
October 2024
Self-supervised pre-training of ViT touch encoders on 460k+ unlabeled DIGIT and GelSight images, tested with frozen features on a new six-task benchmark, TacBench. Sparsh beats task- and sensor-specific models trained end-to-end by 95.1% on average when only 33-50% of the labels are available.
3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile SensingSeminal
October 2024
Thin piezoresistive tactile pads (16x16 per finger) on soft grippers, with tactile readings placed as points next to the camera point cloud and fed to a diffusion policy. Across four bimanual tasks it reaches 80-90% whole-task success, against 45-65% for point-cloud-only policies.
Digitizing Touch with an Artificial Multimodal FingertipSeminal
November 2024
Digit 360 is a finger-shaped vision-based tactile sensor that adds audio, pressure, IMU, heat and gas sensing plus an on-device neural network accelerator. It resolves 7 µm features, predicts normal and shear force with about 1 mN median error, and runs a reflex loop with 1.2 ms event-to-action latency.
Reactive Diffusion Policy: Slow-Fast Visual-Tactile Policy Learning for Contact-Rich ManipulationSeminal
March 2025
A slow latent diffusion policy plans action chunks at 1-2 Hz while a fast decoder adjusts them from tactile or force feedback at 24 Hz, trained on demos from an AR teleoperation system that shows touch. On peeling, wiping and bimanual lifting it beats Diffusion Policy by a wide margin (peeling score 0.95 vs 0.44).
Recent
TactileLab: Sim-to-Real Dexterous Robot Manipulation with Efficient Shear-Sensitive Tactile Simulation
November 2026
TactileLab adds simulated tactile flow, a dense shear field computed from rigid-body contact motion in IsaacLab, so RL policies can feel tangential contact at scale. Trained only in simulation, a LEAP Hand policy using contact depth plus flow makes 1.95 real in-hand rotations in 30 s, against 0.82 with proprioception alone.
Which Physical Tactile Cues Matter for Contact-Rich Imitation Learning? A Controlled Real-Robot Study
November 2026
From one GelSight-style sensor stream they derive tactile RGB, normal/depth maps and marker flow, and compare them as Diffusion Policy inputs over 720 real-robot trials. Touch raises success from 57.8% to 78.9%, and marker flow is the best single cue at 74.4%.
LumiTac: Compact, High-Rate Six-Axis Optical Tactile Sensing for Human-Robot Interaction
November 2026
A camera-free optical tactile sensor, 12 mm across and under 2 g, that reads three forces and three moments from 20 colour light channels for about USD 7. A small MLP reaches R² of 0.966-0.994 per axis at about 300 Hz, and the sensor guides a real gripper through admittance control.
ReflexArc: Bottom-Up Integration of Instantaneous Tactile Reflexes into Whole-Body Humanoid Control
November 2026
A full-body tactile suit on a Unitree G1 triggers whole-body withdrawal reflexes that reach the motors within 10 ms, and the learned whole-body policy is trained to keep balance without cancelling them. Compared with the same reflex delayed by 20 ms, the first force peak drops by 34.1% and the contact impulse by 2.15x.
PanoTac: A Dense Three-Axis Tactile Sensor for Robust Touch-Centric Manipulation
November 2026
A sub-$50 gripper fingertip that fuses a dense piezoresistive array with magnets and magnetometers into a 12x32 map of normal and shear force. With it, visuo-tactile diffusion policies reach 91.7% success on test-tube prep, grain scooping and salt pouring, against 45% for vision alone.
Exploring Proximity and Tactile Sensing for Robotic Manipulation
November 2026
ToFTouch puts three 4x4 multizone time-of-flight depth sensors next to an optical pressure patch on one gripper finger, so the robot can sense objects before and during contact. Closing the gripper early on ToF readings cut grasp cycle time by 7.4%, and ToF depth correction picked items 9/10 times at 2 cm depth errors where picking at the nominal depth failed every time.
ForceLDM: A Force-aware Latent Dynamics Model for Contact-Rich Manipulation
November 2026
ForceLDM distills future force/torque and optical-flow features from a privileged teacher into a π0-based student policy that sees only current observations. On five real-robot contact-rich tasks it averages 87.1% success, 14.9 points above the strongest baseline.
Visuo-tactile Diffusion Policy for Robotic Manipulation with Auxiliary Tactile Dynamics Prediction
November 2026
A visuotactile diffusion policy for simulated UR5 assembly, with a training-only head that predicts how the fingertip images change two steps ahead. The full model reaches 59.0% success against 42.7% for vision only, while adding touch without the auxiliary loss drops to 33.7%.
New papers stay for 14 days.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.