Single-Controller Teleoperation and Multimodal Dataset Generation for a Mobile Manipulator

TL;DR One gamepad drives a mobile base, a Kinova arm and the data recording, and a ROS pipeline filters, synchronizes and exports each demonstration to a Zarr dataset. Four untrained users recorded a bottle pick-and-place task, and the image filter rejected about 1 in 300 frames.
What they do
Collecting demonstrations on a mobile manipulator often means juggling separate controllers for the base and the arm, terminal commands to record, and manual cleanup afterwards. This paper puts all of it behind one DualShock 4 gamepad on a Clearpath Dingo-O base with a Kinova Gen3 Lite arm and a wrist-mounted RealSense D455. A ROS pipeline then turns each recording into a learning-ready Zarr dataset. The authors aim for a lightweight workflow, not a new teleoperation method or learning algorithm.
How it works
- One controller. A button switches the joysticks between the base (forward, sideways, rotation) and the arm, which runs joint by joint (forward kinematics) or in Cartesian space (inverse kinematics). Gains were tuned in MuJoCo first; jitter on hardware went away after switching the Kinova driver to its high-level velocity mode.
- Recording. Another button starts and stops a rosbag of wheel encoders, joint states, end-effector pose and RGB-D images, saved on a laptop over SSH, and then triggers processing.
- Processing. OpenCV drops blurry or dark frames by Laplacian variance and mean brightness, linear interpolation aligns all streams to the camera timestamps, and the result is exported to Zarr plus a cleaned rosbag for replay in RViz or Foxglove.
Results
- In a task where the robot grasps a bottle, drives to a lower surface and places it upright, the operator switched between base and arm modes repeatedly with no significant input delay, and recording needed no external commands.
- Across five trials the quality filter rejected on average 1 in 300 frames at 30 fps; manual checks of kept and rejected frames confirmed it.
- Four users without prior training completed the task and recorded demonstrations.
Limitations
The authors call the evaluation limited and not a formal usability study: one modest task without high-speed or precise motion, and synchronization checked only visually. Longer recordings hit limits in processing time, onboard storage and network bandwidth.
Figures




Abstract
Robot learning systems require synchronized, high-quality demonstrations that combine visual, proprioceptive, and action data. However, collecting such datasets with mobile manipulators is often slowed down by fragmented teleoperation interfaces and manual post-processing workflows. This paper presents an integrated framework for teleoperated multimodal data collection on a Clearpath Dingo-O mobile base equipped with a Kinova Gen3 Lite manipulator and an Intel RealSense RGB-D camera. The framework combines single-controller operation of the base, arm, and recording process with an automated ROS-based pipeline for data capture, image-quality filtering, temporal synchronization, visualization, and export to machine-learning-ready Zarr datasets. We validate the system on a pick-and-place mobile-manipulation task in which the robot transports a bottle between two surfaces. The results show that the unified controller supports complete task execution and recording without external commands, while the data pipeline produces synchronized datasets for short demonstrations and identifies corrupted visual frames. The proposed framework provides a practical basis for producing structured demonstrations for robot-learning research.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
