Ego4D: Around the World in 3,000 Hours of Egocentric Video

Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Ziwei Zhao, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, Jitendra Malik

Seminal October 2021

TL;DR Ego4D is a 3,670-hour dataset of unscripted head-mounted video from 931 camera wearers in 74 locations across 9 countries, with dense narrations and five benchmarks covering the wearer's past, present and future. The baselines leave a lot of room for improvement, e.g. 2.45% Top-5 mAP on short-term object interaction anticipation.

What they do

Most video datasets are short, curated, third-person clips. Ego4D collects long, unscripted, head-mounted video of daily life: 3,670 hours from 931 participants in 74 cities and 9 countries, gathered by 14 teams. On top of it the paper defines five benchmarks for first-person perception and runs baselines on each.

How it works

Collection. Participants wore one of seven head-mounted cameras for 1 to 10 hours at a time; a typical raw clip lasts 8 minutes. Parts of the data add audio, Matterport3D scans, gaze, stereo or synchronized multi-camera views. Wearers gave informed consent; faces in public footage are blurred.
Narrations. Every video gets two independent timestamped narrations: 3.85M sentences, 1,772 verbs and 4,336 nouns, used to build taxonomies and select clips per benchmark.
Benchmarks. Episodic Memory (visual, language and moment queries over past video), Hands and Objects (object state changes), Audio-Visual Diarization, Social Interactions (looking at me, talking to me) and Forecasting (locomotion, hands, next interaction, long-term actions).

Results

Annotation took over 250,000 annotator hours.
Episodic Memory: visual queries reach 42.9% success but 0.13 tAP; language queries 5.80% top-1 recall at tIoU 0.3; moment queries 5.96% mAP.
Hands and Objects: single-frame state-change object detection reaches only 8-14% AP.
Social: 78.07 mAP for looking-at-me, 55.06 for talking-to-me. Short-term interaction anticipation: 2.45% Top-5 mAP on test.

Limitations

74 locations is far from global coverage, wearers live mostly in urban or college-town areas, COVID-19 skewed footage toward stay-at-home activities, and battery life favors the more active parts of the day. Narrations come from annotators at two sites in Africa, so their wording reflects local usage. Each benchmark annotates only 48 to 1,000 of the 3,670 hours.

Figures

From Figure 1: A snapshot of Ego4D (5% of the clips, randomly sampled) showing its spread over 74 locations, activiti... — From Figure 1: A snapshot of Ego4D (5% of the clips, randomly sampled) showing its spread over 74 locations, activities and modalities.

From Figure 3: Scenarios in Ego4D. The outer circle shows the 14 most common scenarios (70% of the data), the word cl... — From Figure 3: Scenarios in Ego4D. The outer circle shows the 14 most common scenarios (70% of the data), the word cloud the remaining 30%, and the inner circle the contributing partner.

From Figure 7: Episodic Memory's three query types over the wearer's past video: visual, language and moments queries.

From Figure 10: The Forecasting tasks: future locomotion, hand movements, next object interactions and sequences of f... — From Figure 10: The Forecasting tasks: future locomotion, hand movements, next object interactions and sequences of future actions.

Abstract

We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite. It offers 3,670 hours of daily-life activity video spanning hundreds of scenarios (household, outdoor, workplace, leisure, etc.) captured by 931 unique camera wearers from 74 worldwide locations and 9 different countries. The approach to collection is designed to uphold rigorous privacy and ethics standards with consenting participants and robust de-identification procedures where relevant. Ego4D dramatically expands the volume of diverse egocentric video footage publicly available to the research community. Portions of the video are accompanied by audio, 3D meshes of the environment, eye gaze, stereo, and/or synchronized videos from multiple egocentric cameras at the same event. Furthermore, we present a host of new benchmark challenges centered around understanding the first-person visual experience in the past (querying an episodic memory), present (analyzing hand-object manipulation, audio-visual conversation, and social interactions), and future (forecasting activities). By publicly sharing this massive annotated dataset and benchmark suite, we aim to push the frontier of first-person perception. Project page: https://ego4d-data.org/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong