ToolDex: An Ego-Exo Human Demonstration Dataset for Creative Physical Tool Use
TL;DR A 60-hour dataset of 10,853 human tool-use demonstrations, often with repurposed tools, recorded by eight RGB-D cameras and a head-mounted stereo camera and annotated with hand pose, 6-DoF tool pose and target-object masks. On its robot-learning benchmark, directly replaying retargeted human motion succeeds 14% of the time on average, against 55% for RL-based tracking.
What they do
ToolDex is a real-world dataset of humans using tools at a table, including creative substitutions such as sweeping nuts with a book instead of a brush. It separates three roles, hand, tool and target object, and many targets are granular or deformable (sand, nuts, dough, sauce). The authors also define benchmarks for VLM tool reasoning, sparse-view 3D/4D reconstruction and robot policy learning.
How it works
- Capture. Eight calibrated Intel RealSense RGB-D cameras (1280x720, 30 Hz) plus a ZED Mini stereo camera on an Apple Vision Pro worn by the demonstrator. Each scene has 4-5 distractor tools.
- Scale. 10,853 trials (60 hours) over 60 (tool, action, target) combinations with 45 tools and 16 action types. Splits hold out a tool, a target, an action or an unseen combination.
- Annotation. FoundationPose runs on every view and the estimates are fused into one 6-DoF tool trajectory. WiLoR MANO hands are scaled to the RGB-D data, SAM2 tracks the target masks, and a joint hand-tool optimization with a contact prior refines both. Every trajectory is checked by hand.
Results
- Robot learning (simulation, six tasks). Kinematic retargeting averages 14.0% success and BC on retargeted data 4.6%. The RL methods ManipTrans and Dex-X reach 40.5% and 55.1%. Collecting sand with a squeegee improves from 12.6% to 57.5% between the two.
- VLM reasoning. The best models include the demonstrated tool in their picks 80% of the time. GPT-6-Astra localizes it with 0.891 box IoU, while Gemini 3.1 Pro scores 0.010.
- Reconstruction. On a held-out camera, InstantSplat + Difix3D reaches 20.45 dB PSNR, 0.5 dB above InstantSplat alone.
Limitations
The data covers indoor tabletop scenes only, and its range of materials is incomplete. Annotations can be noisy under occlusion or heavy contact. The VLM tool-selection metric is deliberately permissive. The policy results come from simulation and are scored on whether the policy tracks the demonstration.
Videos
Figures
Abstract
Creative tool use is a fundamental human capability and an important step toward general-purpose embodied intelligence. It requires agents to reason about object affordances, physical constraints, and manipulation strategies, especially when standard tools are unavailable. However, progress in this area is limited by the lack of real-world datasets that capture human tool-use behavior with sufficient scale, fidelity, and diversity. Existing benchmarks are often simulation-based, focus on simplified interactions, or lack detailed observations and annotations. We introduce ToolDex, a large-scale egocentric-exocentric (ego-exo) human demonstration dataset for creative physical tool use. ToolDex captures scenarios in which humans select, adapt, and repurpose available objects as tools. Using synchronized egocentric cameras and eight exocentric cameras, the dataset provides high-fidelity multi-view observations and rich annotations for hands, tools, target objects, and their deformable interactions. Beyond data collection, we establish a benchmark suite characterizing three core capabilities: physical tool-use reasoning, sparse-view 3D and 4D reconstruction, and robot policy learning. ToolDex provides a foundation for studying human tool-use behavior and developing embodied agents that can adaptively use tools in complex real-world environments. Dataset visualizations and benchmarking results are available at https://tooluse-dataset.github.io/.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.