PCTac: Mechanochromic Photonic-Crystal Tactile Sensor for Dexterous Manipulation
TL;DR A camera-based fingertip whose contact surface is a photonic-crystal film that changes color under load, so the tactile image is a hue map of local stress. Fed to a Diffusion Policy as a hue-difference image, it succeeds in 68 of 90 real-robot rollouts, against 57 for a GelSight Wedge and 29 for vision only.
What they do
GelSight-style sensors image how a gel deforms, and force has to be inferred from that shape. PCTac replaces the gel coating with a mechanochromic photonic-crystal (PC) film: pressing on it changes its periodic microstructure and therefore the color it reflects. The camera image is used directly as the tactile observation of a Diffusion Policy, with no force or geometry reconstruction.
How it works
- Fingertip. A wedge-shaped housing (after GelSight Wedge) with an OV5647 camera, a mirror that folds the optical path, and white LEDs only. The contact stack is black silicone ink, a 20 x 30 mm PC film 15 µm thick, and a 3 mm Shore A 15 elastomer on acrylic.
- Hue representation. Each image is converted to HSV and compared with an unloaded reference; the hue difference is encoded as sine and cosine and masked by saturation.
- Policy. SO-101 leader-follower arms, one PCTac finger, wrist and global cameras. ResNet-18 encoders feed a Diffusion Policy trained on 60 demonstrations per task.
Results
- Hue changes with normal load over 22 indentation trials, with hysteresis between loading and unloading. With a 5 mm spherical indenter the response saturates near 1.5 N.
- Over 90 rollouts (30 per task): PCTac Hue 68 (75.6%), PCTac RGB 60 (66.7%), Wedge RGB 57 (63.3%), vision only 29 (32.2%).
- Per task, PCTac Hue gets 24/30 on whiteboard wiping, 28/30 on test-tube insertion and 16/30 on perturbed insertion; Wedge RGB gets 23, 21 and 13.
Limitations
The elastomer conforms poorly to concave features, which give weaker responses than raised ones. The film responds to local stress, so different mixes of normal and shear load can produce similar color patterns. Most policy failures were misgrasps and misinsertions before contact.
Videos
Figures
Abstract
Modern visuotactile sensors primarily sense contact by imaging the deformation of a compliant interface. Using techniques such as photometric stereo, they can recover detailed contact geometry, often represented as surface normals or depth. However, force-related information typically still needs to be inferred indirectly through calibration, geometric reconstruction, or learned mappings. Mechanochromic photonic-crystal (PC) materials offer an alternative physical encoding mechanism: contact loading changes their periodic microstructure and spectral response, directly transforming local stress into a spatially resolved structural-color field. Despite this potential, PC materials have rarely been explored as visuotactile sensors for closed-loop robot learning. We present PCTac, an open-source mechanochromic PC fingertip that uses a dense hue field as its tactile observation. Loading characterization demonstrates a repeatable relationship between the applied normal load and the observed hue response. We further integrate this representation directly into a Diffusion Policy and evaluate it on three real-robot manipulation tasks. Across 90 physical rollouts, PCTac Hue achieves 68 successes (75.6\%), compared with 60 (66.7\%) for PCTac RGB, 57 (63.3\%) for Wedge RGB, and 29 (32.2\%) for vision-only. PCTac Hue also achieves the highest observed success rate on each evaluated task, including Perturbed Test-Tube Insertion, where it achieves 16/30 successes versus 13/30 for tactile RGB baselines and 1/30 for vision-only. These results demonstrate the potential of mechanochromic structural color as a compact, load-sensitive tactile representation for robot learning and closed-loop manipulation. Project page: https://bey0ndt1me.github.io/PCTac-page/.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.