← Paper Digest

Single-View RGB-D Force--Deformation Estimation for Robotic Manipulation of Elasto-Plastic Materials

Dezhi Sun, Marijn van Hilten, Kenan Niu

November 2026

TL;DR A Franka FR3 presses into Play-Doh while a wrist-mounted RGB-D camera measures the dent it leaves. A small MLP maps the estimated dent volume back to the robot's own force estimate with a held-out RMSE of 1.857 N (R² = 0.440).

What they do

Elasto-plastic materials such as clay and dough keep their deformation, so every push changes the shape the robot sees next. This paper measures how a robot’s intrinsic force estimate relates to the persistent dent it leaves, using only a standard arm and one wrist-mounted RGB-D camera, with no external force sensor. The scope is controlled normal indentation of one Play-Doh-type material.

How it works

  • Platform. A 7-DoF Franka FR3 estimates end-effector force from its joint torques. An Intel RealSense L515 rides on the end-effector, and four ArUco markers define the region of interest.
  • Indentation. Each trial scans the undeformed surface, pushes down under position-impedance control until the filtered force reaches the target, returns to the same viewpoint and scans again.
  • Dent volume. The clouds are cropped, the background plane is removed and a reference plane is fitted around the known contact. The volume below that plane is the estimated displaced volume. For a sequence of pushes, ICP alignment, signed surface distances and DBSCAN clustering find the dominant affected region.
  • Force regression. A quadratic fit and a 1-32-32-1 MLP map volume to force.

Results

  • Over 27 indentations from 5 to 25 N, displacement is hard to separate from surface variation up to about 10 N and becomes clearer from about 12.5 N.
  • On held-out data the MLP reaches RMSE 1.857 N and R² = 0.440. On a separate set, the quadratic model has lower error at 16 N and the MLP at 19 and 23.5 N.
  • Five 16 N trials reach the same stopping force along different paths: x varies by up to about 6 mm and final penetration depth by about 5 mm.
  • In an eight-push sequence at 4 N per push, a dominant affected region is found from the second push onward.

Limitations

One material under controlled lab conditions, and the MLP covers a narrow force range. The single-view volume misses self-occluded or laterally displaced material, and neighboring dents merge into one region. Closed-loop shaping toward a target is not tested.

Figures

From Figure 1: The robotic platform: a Franka FR3 with an eye-in-hand RealSense L515 RGB-D camera above a Play-Doh-ty...
From Figure 1: The robotic platform: a Franka FR3 with an eye-in-hand RealSense L515 RGB-D camera above a Play-Doh-type specimen, with ArUco markers defining the region of interest.
From Figure 1: The deformation-estimation pipeline: robot push, deformed point cloud, surface segmentation from two v...
From Figure 1: The deformation-estimation pipeline: robot push, deformed point cloud, surface segmentation from two views, dent point extraction, and reference-plane fitting to estimate the displaced volume.
From Figure 3: Measured force versus estimated displaced volume over 27 indentations, with the quadratic fit and its ...
From Figure 3: Measured force versus estimated displaced volume over 27 indentations, with the quadratic fit and its 95% prediction band.
From Figure 3: MLP force predictions on the held-out test subset against ground-truth force (R² = 0.440, RMSE 1.857 N).
From Figure 3: MLP force predictions on the held-out test subset against ground-truth force (R² = 0.440, RMSE 1.857 N).
Abstract

Persistent deformation complicates elasto-plastic manipulation by coupling physical interaction with spatially distributed shape changes. We investigate the force--deformation relationship using a standard robot, a single eye-in-hand RGB-D camera, and intrinsic force estimates. A 7-DoF Franka Emika FR3 performs normal indentations while an Intel RealSense L515 captures pre- and post-contact surfaces. Point-cloud processing localizes deformation and estimates displaced volume. Force--volume characterization includes 27 measurements over nine force levels. A compact multilayer perceptron (MLP), trained on 52 force--volume pairs at four levels, achieves a held-out RMSE of 1.857 N and $R^2=0.440$. A separate comparison shows lower error for a quadratic model at 16 N and for the MLP at 19 and 23.5 N. An eight-step experiment at 4 N per interaction demonstrates cumulative deformation localization from the second indentation onward. These results establish a practical measurement basis for deformation-aware manipulation using intrinsic force estimates and single-view geometry.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.