Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Embedded Sim-to-Real Visual Servoing: Learned Reaching and Force-Guided Placement

Dezhi Sun, Duy Khang Banh, Kenan Niu

November 2026

TL;DR A Jetson-driven Franka FR3 picks a block and inserts it into a shape sorter, either with two small PPO reaching policies trained in simulation or with a scripted force-guided spiral search. Over 50 real trials each, the learned hybrid completed 70% and the force-guided pipeline 74%.

What they do

They build one embedded pipeline for a pick-and-insert task on a Franka Research 3. A wrist RealSense D435i and a Jetson AGX Orin turn a segmented object or hole into a metric target in the robot base frame. On top of that shared front end they compare two ways to move: small PPO reaching policies trained in simulation and run with fixed weights, and a scripted Cartesian sequence that uses contact force to find the hole.

How it works

  • Perception. YOLO11-seg (TensorRT, trained on about 100 labeled images of four shapes) gives a mask. Depth at the mask centroid is back-projected and moved into the base frame with Tsai-Lenz eye-in-hand calibration, and the target updates as the wrist camera moves.
  • Learned hybrid. Two PPO policies trained in RLBench for 10M steps each: Policy A reaches the object, Policy B the hole region. Each sees a 15-number state (joints, end-effector position, class, target, one orientation term) and outputs joint increments. The actor-critic pair has 10,895 parameters. Grasp, lift and release are scripted.
  • Force-guided analytical. Scripted approach, grasp and transport, then a descent to about 2 N normal contact and a planar spiral search. A drop in height while force relaxes signals entry and triggers release.
  • Both branches run through the same Cartesian impedance controller.

Results

  • Simulation, 1000 episodes each: Policy A 87.2% success (5.2% collisions), Policy B 83.1% (no collisions, 16.9% timeouts).
  • Real robot, 50 trials each: learned hybrid 35/50 (70.0%), force-guided 37/50 (74.0%). The 95% confidence intervals overlap (56.2-80.9% vs 60.4-84.1%).
  • Segmentation macro mAP50 is 0.953. Perception takes 34.55 ms per frame on the Jetson (about 28.9 FPS), with postprocessing the largest share (15.3 ms).

Limitations

The physical study uses one square object roughly aligned with its hole and reports only end-to-end completion, not per-stage failures. Policy B never sees force or insertion state. The authors name gripper occlusion, detector false positives that make the target jump, and depth or calibration error as failure mechanisms, but did not measure how often each occurs.

Figures

From Figure 1: (a) The sorting workspace and the wrist-mounted D435i and gripper on the FR3. (b) Two PPO policies are...
From Figure 1: (a) The sorting workspace and the wrist-mounted D435i and gripper on the FR3. (b) Two PPO policies are trained in RLBench and transferred with fixed weights; on the robot, Jetson-based RGB-D perception builds one base-frame target used by either PPO reaching or force-guided placement, both run through Cartesian impedance control.
From Figure 2: Motion stages of the two pipelines. The learned hybrid (blue) uses PPO policies for object and hole-re...
From Figure 2: Motion stages of the two pipelines. The learned hybrid (blue) uses PPO policies for object and hole-region reaching with scripted handling in between; the analytical pipeline (orange) adds about 2 N normal-force regulation and spiral search after contact.
From Figure 3: (a) Simulation outcomes over 1000 episodes per policy. (b) End-to-end outcomes over 50 real trials per...
From Figure 3: (a) Simulation outcomes over 1000 episodes per policy. (b) End-to-end outcomes over 50 real trials per strategy. (c) Mean and 95th-percentile TensorRT perception latency over 1000 frames.
Abstract

Embedded perception-to-control systems must combine visual targeting with reliable contact handling under tight computational constraints. We present a shared architecture on an NVIDIA Jetson AGX Orin controlling a Franka Research 3 through Cartesian impedance control. RealSense D435i observations, YOLO11-seg segmentation, depth back-projection, and hand--eye calibration provide metric robot-frame targets. We compare two strategies: compact proximal policy optimization (PPO) policies transferred from simulation for object- and hole-region reaching, and force-guided Cartesian placement using approximately 2 N normal contact and spiral search. The PPO policies achieved 87.2% and 83.1% success over 1000 simulation episodes each. Across 50 real-robot trials per strategy, the learned hybrid and force-guided analytical pipelines achieved 70.0% and 74.0% completion, respectively. Mean perception latency with TensorRT was 34.55 ms. These results support an embedded architecture combining learned free-space reaching with explicit contact correction.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.