A human-in-the-loop Control Paradigm for Construction Robotics: From VR-Guided Teleoperation to Semi-Autonomous Dual-Arm Operation

Jinhua Chen, Xiaoji Xu, Zhongyuan Liao, Huang Zhenmin, Zhang Ye, Changying Xiang

November 2026

TL;DR A dual-arm robot drills concrete, places brackets and fastens screws under VR teleoperation, rule-based autonomy, or shared control in which the operator picks targets and the robot aligns and makes contact. Shared control raised novices' bracket-placement success from 4% to 81% over teleoperation; autonomy did best in nominal conditions but fell to 33% under 75% occlusion, where shared control kept 73%.

What they do

Construction tasks such as drilling a facade and fastening brackets are contact-rich and precise, which makes pure teleoperation hard and full autonomy brittle. This system lets an operator move along a spectrum: full VR teleoperation of two arms, rule-based autonomy, or shared control in which each skill primitive is assigned to the human or the robot. They compare all three on drilling, bracket pick-and-place and dual-arm screw fastening in a lab mock-up.

How it works

Hardware. Two 6-DoF FAIRINO FR5 arms with a gripper, drill and screwdriver, a wrist force/torque sensor, a RealSense D435 and a Meta Quest 3.
Skills. Each task is a fixed sequence of skills (e.g. target selection, coarse approach, fine alignment, drilling, retraction), run by a state machine. In shared control the human picks targets and makes coarse motions; the robot handles fine alignment, flipping the arm around and the drilling or fastening itself.
Perception and contact. A fine-tuned YOLO segments holes and marks; RANSAC fits the facade plane so tools go in along its normal. Drilling uses a 30 N force-controlled feed.
Priority rules. E-stop beats force limits, which beat operator override, which beats the autonomous controller.

Results

Ten novice operators, 100 trials per mode and task; autonomy ran 100 unmanned repetitions.

Success (teleop / shared / auto): drilling 53 / 82 / 96%, bracket placement 4 / 81 / 97%, screw fastening 3 / 68 / 93%.
Shared-control drilling took 38.7 s against 92.5 s for teleoperation; NASA-TLX workload fell from 61.7 to 32.9.
Under disturbances autonomy fell to 63% in dynamic colored lighting, 33% at 75% occlusion and 42% with other moving objects; shared control stayed at 76%, 73% and 74%.

Limitations

Lab mock-up only, not at height. The skills are hand-written rules. Robustness and workload were tested only for drilling, and there are no ablations of individual components.

Figures

From Figure 1: Overview of the system: VR interface, decision and control center, perception with a RealSense camera,... — From Figure 1: Overview of the system: VR interface, decision and control center, perception with a RealSense camera, and the dual-arm execution layer, moving from full teleoperation to semi-autonomous control to full autonomy.

From Figure 3: Three control layers, from the human-dominant intent layer through the shared-autonomy action layer to... — From Figure 3: Three control layers, from the human-dominant intent layer through the shared-autonomy action layer to the robot-dominant low-level control layer.

From Figure 9: The experimental setup for the bracket task: two arms, a concrete plate, a screw holder, a bracket and... — From Figure 9: The experimental setup for the bracket task: two arms, a concrete plate, a screw holder, a bracket and the gripper and motor tools.

From Figure 12: Holes drilled autonomously versus by teleoperation, each labeled with depth, perpendicularity and pos... — From Figure 12: Holes drilled autonomously versus by teleoperation, each labeled with depth, perpendicularity and position error against a 50 mm target depth. Autonomous holes are more accurate and consistent; teleoperated ones are often misaligned or too shallow.

Abstract

Construction tasks in hazardous and unstructured environments require both human adaptability and robotic precision, yet existing approaches often treat teleoperation and autonomy as separate control modes. This paper presents a human-in-the-loop control paradigm that enables structured transitions among VR-guided teleoperation, semi-autonomous shared control, and rule-based autonomous execution for contact-rich dual-arm construction tasks. The proposed system combines immersive VR guidance, a manually defined library of reusable skill primitives, and a shared-autonomy framework with explicit human-robot role allocation and command-priority rules. This integration allows operators to retain supervisory and corrective authority while delegating repetitive and precision-critical actions to the robot. The paradigm is validated on a dual-arm platform through representative facade-installation tasks. Results show that autonomous execution provides the best performance under well-calibrated nominal conditions, whereas semi-autonomous control improves resilience when environmental disturbances exceed the predefined autonomous pipeline's correction capability. Semi-autonomous control also substantially reduces subjective operator workload compared with full teleoperation. These findings demonstrate that effective construction-robot control should adapt human involvement to task and environmental uncertainty, using autonomy for repeatable operations and shared control for disturbance recovery.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong