What is First-Person View Data?
Video and sensor data captured from the operator's viewpoint with wearable devices, used for embodied AI and action understanding.
First-person view data is video and sensor data captured from the operator's viewpoint with a wearable device. The viewpoint is the defining property: the camera sees what the wearer's hands are doing, which is exactly the information third-person footage loses.
The capture conditions are difficult. Footage is shaky, the field of view is limited, and the rig's comfort determines how long a session can run. Those constraints shape both the annotation and the schedule.
The uses are embodied AI, action understanding, and procedure learning. All three benefit from the hand and gaze channels, which are what a head-mounted rig can record and a fixed camera cannot.
What buyers get wrong
First-person footage is shaky and has a limited field of view, which makes annotation markedly harder than third-person. How comfortable the capture rig is to wear also directly limits how long sessions can run.
The specification detail that decides everything
Whether hand keypoints, gaze tracking, and inertial measurement data are included.
This is the line item that most often separates a dataset that works from one that gets re-ordered. It belongs in the specification before production starts, not in the delivery review.
How we source it
We do not hold inventory in this category. A requirement comes in, we match it to a producer who can meet the specification, and we agree the quality bar with both sides before recording begins. A pilot batch goes first so misalignments surface early.
Scenarios this covers
Buyers rarely ask for a category by name. They describe a situation, and the situation is what tells us whether this category is the right one. If yours is in the list, name it in the brief and it goes straight into the specification. If it is not, describe it anyway, because it usually means the requirement sits in a neighbouring category.
What good First-Person View Data looks like
First-person data is judged on the sensor channels and on whether the capture conditions are documented.
-
Hand keypoints and gaze recorded where the specification needs them
These are the channels that make first-person data worth the capture difficulty, and a vision-only recording loses most of the advantage.
-
Inertial measurement included and synchronized
Head motion is what makes the footage shaky, and the inertial channel is what allows the motion to be separated from the scene.
-
Session duration documented
How long the wearer could comfortably record determines the sequence length, and the limit is a property of the rig rather than of the task.
-
Shake and motion blur characterized
The proportion of frames affected, because a corpus that does not state it will be evaluated on footage cleaner than it is.
-
Wearer metadata
Who the operator was and their familiarity with the task, because expertise changes the motion pattern substantially.
Public datasets in this category
These are openly documented resources, not products we sell. Each one covers part of the category, and knowing where each stops is usually what decides whether a project needs a collection of its own.
-
Ego4D
Thousands of hours of first-person video from hundreds of wearers across many countries, with benchmark tasks defined on it. The widest geographic spread in this category.
-
EPIC-KITCHENS
First-person footage of kitchen activity with fine-grained action and object annotation. The annotation depth is the contribution, and the setting is one domain.
-
Ego-Exo4D
Paired first-person and third-person recordings of the same activity, with expert commentary. The pairing is what makes it possible to study what the first-person view adds over the third-person view.
-
Charades
Scripted activity videos with action labels, partly first-person. Useful for action understanding and the actions are performed rather than natural.
Questions to ask a supplier
Questions that establish which sensor channels are present and how the capture constraints were handled.
- Are hand keypoints and gaze tracking recorded, or is it vision only?
- Is inertial measurement included, and is it synchronized with the video?
- What is the typical and maximum session duration?
- What proportion of frames are affected by shake or motion blur?
- Is wearer expertise recorded?
The vague answer in this category is about the wearer. A supplier who says the corpus is first-person without describing who wore the camera and what they were doing has described a viewpoint rather than a dataset. Expertise, familiarity with the environment and the task itself all change the motion, and none of them are visible in the video.
A verifiable answer names the wearers, their expertise level, the sensor set and the synchronisation method. Ask how many sessions were discarded for camera displacement, because a head-mounted camera moves against the head, and an unmeasured shift makes the gaze labels point at the wrong place.
Technical attributes
The items below are the ones a specification can hold a producer to, and the ones that decide whether a delivery is usable for training rather than only present.
| Sensor channels | Video, hand keypoints, gaze, inertial, listed |
|---|---|
| Synchronization | Channels aligned on one clock, verified |
| Session duration | Typical and maximum, stated |
| Motion artifact rate | Proportion of affected frames |
| Wearer metadata | Expertise and task familiarity recorded |
| Delivery format | Synchronized channels with a per-session manifest |
The sensor synchronisation column decides whether the corpus can be used at all, and it is the one most often assumed rather than measured. Video, gaze and inertial streams drift against each other over a long session. Without a verified sync, the hand and gaze labels point at the wrong frame and the error is systematic rather than random.
The values come from the application. Assistive systems, robot learning from demonstration and activity recognition need different sensor sets and different session lengths. The specification should name the application, because a first-person corpus built for one of them is largely unusable for the others.
A mistake here is measured as a model that cannot localise attention. Where the sync or the camera displacement is wrong, the gaze labels are offset and the model learns the offset. The error is consistent, which means it is learned rather than averaged away, and it cannot be corrected after delivery.
How First-Person View Data is collected
The rig and the session design are the constraints, and the collection is built around them.
- Rig selection and comfort testing. The rig is chosen for the sensor set the specification needs, and session length is set by what a wearer can tolerate rather than by the task.
- Task selection for the sensor set. Tasks are chosen where the first-person viewpoint adds information, since a task a fixed camera records well does not need this capture.
- Synchronized multi-channel recording. Video, hand, gaze and inertial channels are logged on one clock, and the synchronization is verified rather than assumed.
- Motion artifact characterization. The proportion of frames affected by shake or blur is measured and reported, because it determines how much of the corpus is usable.
- Session metadata and delivery. Wearer, task, duration and rig are recorded per session and delivered with the data.
After collection, the sensor streams are synchronised and verified, the wearer and task metadata are attached, and the files are packaged with the calibration for each session. The sync verification is the step that makes the rest of the annotation meaningful.
The pilot for first-person capture is a short session per wearer with the full sensor set running, because the property being tested is whether the streams stay synchronised over a session and whether the camera displacement can be measured. A few sessions show the drift and the discarded-session rate, and both figures set the plan for the full collection.
Acceptance is checked on the verified synchronisation and on the per-session calibration, not on the apparent quality of the video. The specification should name the wearers and the sensor set before recording starts. Where the pilot shows the displacement cannot be measured, the mounting is changed rather than the gaze labels accepted, since an unmeasured shift makes every gaze label point at the wrong place.
How a delivery goes wrong
The defects below are the ones that recur across this category. Each is decided at collection or annotation time, which is why none of them can be repaired in delivery review.
-
Sensor channels are out of sync
Video, gaze and inertial streams drift against each other during long sessions. Without a verified synchronization, the hand and gaze labels point at the wrong frame and the error is systematic.
-
Wearer expertise was not recorded
An expert and a novice perform the same task with visibly different motion. Without an expertise label the corpus cannot be stratified, and the variance reads as noise rather than as a variable.
-
Only successful attempts were kept
Manipulation fails often and the failures are informative. A corpus of clean completions teaches a model that the task always proceeds as planned.
Why this matters to buyers
The first-person viewpoint is where the information about manipulation lives. A third-person camera sees the object and not the hand's relationship to it, and for procedure learning that relationship is the signal.
The second driver is that the capture constraints are real and have to be planned around. Rig comfort sets session length, which sets sequence length, which sets what the corpus can train. That chain is designed rather than discovered.
A first-person model trained on data with a sync error learns the error. It performs well on the corpus and fails on any new recording, and the failure is hard to diagnose because the labels look correct in isolation. The corpus cannot be repaired, since the offset was never measured.
The second cost is the wearer pool. A corpus collected from a handful of wearers encodes their motion and their environments, and a model trained on it transfers narrowly. Widening the pool means a new collection with new recruitment, which in this category is slow because the recording sessions are long.
A first-person corpus with verified synchronisation gives a model whose gaze and hand labels point at the right frame. The team avoids a calibration pass it cannot run on its own, and the wearer and task metadata mean the corpus can be filtered to the situations the product will meet.
The second gain is the wearer record. Motion and environment are encoded by who wore the camera, and a delivery that records the wearers and their expertise can be extended with new ones without the pool being re-derived. That record is what lets a small corpus grow into a varied one.
Specification at a glance
| Category | First-Person View Data |
|---|---|
| Group | Non-speech data |
| Critical spec | Whether hand keypoints, gaze tracking, and inertial measurement data are included. |
| Included by default | Pilot batch, metadata schema, source and consent documentation |
| Pricing | Quoted from specification. No published rates |
Related categories
-
LLM Training Data
Text corpora for LLM pretraining and fine-tuning: instruction data, preference data, and multi-turn dialogue data.
-
Computer Vision Data
Annotated image and video data, with annotation types including classification, bounding boxes, segmentation masks, and keypoints.
-
NLP Data
Annotated text for NLP tasks: named entities, dependency syntax, sentiment polarity, and relation extraction.
-
Multilingual Language Data
Parallel or non-parallel text corpora spanning multiple languages, used for multilingual models and low-resource language research.
-
Translation Data
Parallel corpora pairing source and target language, used for machine translation training and evaluation.
-
Dictionary Data
Structured dictionary entries with definitions, parts of speech, pronunciation, and example sentences, used for dictionary products and lexical understanding tasks.
Questions about First-Person View Data
Why is first-person footage harder to annotate?
Because it is shaky and the field of view is limited. The annotator cannot always see what is being manipulated, and the motion makes tracking harder. That is why agreement is measured specifically on the hand and object labels.
Is gaze tracking necessary?
Only where the specification needs it. Gaze is informative for procedure learning and attention modelling, and it adds a calibration step and a failure mode. Where it is not needed, the capture is simpler without it.
How long can a session run?
It is limited by rig comfort rather than by the task, and the limit is stated per rig. That limit is what determines the maximum sequence length the corpus can offer.
Do we need inertial data?
It is strongly recommended. Head motion is what makes the footage shaky, and the inertial channel is what allows that motion to be separated from the scene rather than only tolerated.
Do you have First-Person View Data available now?
No. We do not hold inventory. Everything is produced against a specification. That means the timeline starts when the spec is agreed rather than immediately, and it also means the data matches your requirement instead of approximately matching something already built.
Can this be combined with other categories in one delivery?
Yes, and it is common. A single specification can cover several categories across several languages, with one pilot batch and one quality bar. It usually reduces total cost because contributor recruitment and project setup are shared.
What does the consent documentation cover?
Signed contributor consent stating the intended use, the collection methodology, and the annotation guideline the data was produced under. For projects involving EU data subjects the chain is built to satisfy GDPR, including the transfer mechanism. The terms for moving the data across a border.
Request First-Person View Data
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.