Interactive Perception of Granular Materials via Robotic Scooping and Force-Torque Sensing

Samuli Hynninen, Mohammad Amman, Meri Mäkelä, Anirvan Dutta, Ville Kyrki

November 2026

TL;DR A Franka scoops granular materials with a spoon on a wrist force-torque sensor, and Latent Filter++ estimates grain size and bulk density online from the raw signal. On materials left out of training it beats a supervised CNN+LSTM (grain-size error 0.50 vs 0.63 in log10 with five of 26 materials held out).

What they do

How a robot should carry a spoonful of rice or flour depends on properties such as grain size and density. This work infers them by touch. A Franka Panda scoops material with a spoon mounted on a 6-axis ATI force-torque sensor. Latent Filter++, a causal deep state-space model, updates its estimate as the raw F/T signal arrives, with no hand-crafted preprocessing. The dataset has 26 everyday materials and 1316 scoops.

How it works

Latent state. As in the earlier Latent Filter, the latent splits into a fast state recoverable from one observation and a material state, constant within a scoop, that an LSTM updates over time.
Training changes. The LSTM also receives the innovation, the gap between predicted and observed state. Random observation masking forces the model to rely on its learned dynamics.
Encoder. Inspired by slow- and fast-adapting mechanoreceptors, learnable band-pass filters per F/T axis and learnable temporal windows replace a generic CNN. A matching decoder predicts per-axis means and per-band energies.
Readout. Kernel ridge regression maps the frozen latent to each property, so a few scoops of a new material can adapt it without retraining.

Results

Errors are in log10 (0.30 is a factor of two). With all materials in training, grain-size and density errors at 13.8 s are 0.22 and 0.07, vs 0.35 and 0.13 for the prior latent filter.
With five materials held out, it beats a supervised CNN+LSTM: 0.50 vs 0.63 for grain size, 0.17 vs 0.20 for density. Adding scoops of a held-out material lowers leave-one-out grain-size error from 0.46 (one scoop) to 0.23 (forty).
For material classification from a 0.14 s window, the encoder reaches 52.6%, vs 48.1% for a CNN and 39.7% for GmClass.
On 12 materials, the maximum acceleration before spilling correlates with grain size (r_s = 0.74) and weakly with density (0.21).

Limitations

The model does not extrapolate beyond the range of materials seen in training. Its largest leave-one-out errors are on materials near the edges of the learned representation, such as powdered sugar and potting gravel. Grain size is noisier than density, and angle of repose is poorly estimated. Few-shot adaptation needs manually chosen anchor scoops. The transport test reports correlations only; no controller uses the estimates.

Figures

From Figure 1: Problem setup. A Franka arm with an ATI F/T sensor scoops granular material, Latent Filter++ processes... — From Figure 1: Problem setup. A Franka arm with an ATI F/T sensor scoops granular material, Latent Filter++ processes 0.25 s windows of raw F/T signal, and the inferred grain size and density are meant to guide manipulation without spilling.

From Figure 2: The Latent Filter++ architecture for unsupervised deep state-space learning and inference: a filter-ba... — From Figure 2: The Latent Filter++ architecture for unsupervised deep state-space learning and inference: a filter-bank encoder, an LSTM and interaction-dynamics model fused by Bayes rule, and a sub-band decoder.

From Figure 3: The 26 granular materials used in the experiments, from big macaroni and peanuts to wheat flour and po... — From Figure 3: The 26 granular materials used in the experiments, from big macaroni and peanuts to wheat flour and powdered sugar.

From Figure 6: Error of the predicted grain size and bulk density over the course of a scoop for Latent Filter++, the... — From Figure 6: Error of the predicted grain size and bulk density over the course of a scoop for Latent Filter++, the prior latent-filter approach and a supervised two-layer CNN+LSTM.

Abstract

Granular materials exhibit complex and highly variable interaction dynamics, making their reliable manipulation challenging without knowledge of their physical properties. We present an interactive perception framework that infers granular material properties directly from force-torque measurements acquired during scooping. The proposed Latent Filter++ combines a bio-inspired frequency-temporal filter-bank representation with a causal latent dynamical model, allowing the representation to evolve as interaction observations become available while avoiding hand-crafted signal preprocessing. The learned latent representation captures physically meaningful variation in granular materials and supports estimation of properties including grain size and bulk density. Evaluated on 26 diverse granular materials, the approach generalizes to materials excluded from training and enables few-shot adaptation of property-specific mappings using a small number of additional interaction samples. Finally, we demonstrate the relevance of the inferred properties to downstream manipulation by examining their correlation with the maximum acceleration the materials can withstand without spilling from the spoon during transport.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong