Single-View RGB-D Force--Deformation Estimation for Robotic Manipulation of Elasto-Plastic Materials

TL;DR A Franka FR3 presses into Play-Doh while a wrist-mounted RGB-D camera measures the dent it leaves. A small MLP maps the estimated dent volume back to the robot's own force estimate with a held-out RMSE of 1.857 N (R² = 0.440).
What they do
Elasto-plastic materials such as clay and dough keep their deformation, so every push changes the shape the robot sees next. This paper measures how a robot’s intrinsic force estimate relates to the persistent dent it leaves, using only a standard arm and one wrist-mounted RGB-D camera, with no external force sensor. The scope is controlled normal indentation of one Play-Doh-type material.
How it works
- Platform. A 7-DoF Franka FR3 estimates end-effector force from its joint torques. An Intel RealSense L515 rides on the end-effector, and four ArUco markers define the region of interest.
- Indentation. Each trial scans the undeformed surface, pushes down under position-impedance control until the filtered force reaches the target, returns to the same viewpoint and scans again.
- Dent volume. The clouds are cropped, the background plane is removed and a reference plane is fitted around the known contact. The volume below that plane is the estimated displaced volume. For a sequence of pushes, ICP alignment, signed surface distances and DBSCAN clustering find the dominant affected region.
- Force regression. A quadratic fit and a 1-32-32-1 MLP map volume to force.
Results
- Over 27 indentations from 5 to 25 N, displacement is hard to separate from surface variation up to about 10 N and becomes clearer from about 12.5 N.
- On held-out data the MLP reaches RMSE 1.857 N and R² = 0.440. On a separate set, the quadratic model has lower error at 16 N and the MLP at 19 and 23.5 N.
- Five 16 N trials reach the same stopping force along different paths: x varies by up to about 6 mm and final penetration depth by about 5 mm.
- In an eight-push sequence at 4 N per push, a dominant affected region is found from the second push onward.
Limitations
One material under controlled lab conditions, and the MLP covers a narrow force range. The single-view volume misses self-occluded or laterally displaced material, and neighboring dents merge into one region. Closed-loop shaping toward a target is not tested.
Figures




Abstract
Persistent deformation complicates elasto-plastic manipulation by coupling physical interaction with spatially distributed shape changes. We investigate the force--deformation relationship using a standard robot, a single eye-in-hand RGB-D camera, and intrinsic force estimates. A 7-DoF Franka Emika FR3 performs normal indentations while an Intel RealSense L515 captures pre- and post-contact surfaces. Point-cloud processing localizes deformation and estimates displaced volume. Force--volume characterization includes 27 measurements over nine force levels. A compact multilayer perceptron (MLP), trained on 52 force--volume pairs at four levels, achieves a held-out RMSE of 1.857 N and $R^2=0.440$. A separate comparison shows lower error for a quadratic model at 16 N and for the MLP at 19 and 23.5 N. An eight-step experiment at 4 N per interaction demonstrates cumulative deformation localization from the second indentation onward. These results establish a practical measurement basis for deformation-aware manipulation using intrinsic force estimates and single-view geometry.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
