What is Egocentric Video Data?

The camera is the easy part (a head-mounted action camera at 30 to 60 fps), and the annotation is what costs. The labels that matter are hand-object interaction: when a hand enters frame, what it touches, what action it performs. Ego4D, the largest public set, holds about 3,670 hours from 931 camera wearers across 74 locations. EPIC-KITCHENS-100 is smaller and deeper: 100 hours in 45 kitchens, with roughly 90,000 annotated action segments.

What separates egocentric capture from other video is that it records life as it happens, which means it records people who did not agree to appear. Ego4D was collected under a documented de-identification procedure, and commissioned capture needs the same: a written rule for bystanders, applied before delivery.

For robot learning the value is observation without a robot (no platform, no operator, no resets between attempts), which is why egocentric footage scales to hours that teleoperation cannot reach. Capture for a specific task, environment or hand-object domain can be commissioned to order, and the bystander policy belongs in the brief, because it is far cheaper to decide before recording than after.

Types of egocentric capture

Egocentric capture divides into scripted and naturalistic. A scripted recording works through a task list (cooking a set of dishes, assembling parts), so the chosen tasks are covered and the footage is dense with the behavior of interest. Naturalistic capture records daily activity as it happens, buying variety no task list predicts and costing control: the behaviors you need may not occur.

The annotation layer is the second split, and where the cost sits. Raw footage is one product; action segments with start and end times are another; hand-object interaction labels (when a hand enters frame, what it touches, what it does) are the deepest and most expensive. Some sets add narration from the wearer, cheap to record and useful for language-conditioned training.

The rig is the third variation. A single head-mounted camera is the standard; its field of view decides whether the hands stay visible during close work: exactly where the labels are needed. Multi-camera rigs widen the coverage at the cost of synchronization and bulk. Wearer count is the last axis: many wearers spread the behavior across hands and techniques; one gives depth on a single person.

What to specify when buying

Specify the environments and the wearers. A kitchen, a warehouse and a lab produce different hand-object interactions, and a set collected in one does not cover the others. The number of distinct wearers matters as much as the hours, because hand appearance, technique and speed vary from person to person.

The bystander policy belongs in the brief. A head-mounted camera records people who never agreed to appear (faces, voices, screens, badges, documents), and each of those has a different removal cost. Blurring a face is routine; a legible screen or a spoken name is harder, and a wearer narrating what they see can describe people who were never filmed.

Then the annotation scope. Which labels, at what granularity, under what guideline, and with what quality check. An annotated hour costs a multiple of a raw hour, and the multiple follows from the label depth rather than the footage. If video and labels are delivered together, say how they must stay aligned, since drift between the two is expensive to find late.