What is Video Data?
Annotated video footage, with the annotation needed for tasks such as action recognition, object tracking, and temporal segmentation.
Video data is footage with annotations for action recognition, object tracking, or temporal segmentation. The annotation unit is what distinguishes the subtypes, and it determines the cost more than any other decision.
Two properties drive the economics. Storage and transfer are expensive at video scale, and annotation takes several times the hours of image work because the annotator has to watch the footage. Project schedules therefore have to be built on frame count rather than clip count.
The uses are action recognition, which labels what happens in a clip; tracking, which follows objects across frames; and temporal segmentation, which marks where events begin and end. Each needs a different annotation and a different guideline.
What buyers get wrong
Storage and transfer costs for video are high, and annotation takes several times the hours of image work. Project schedules have to be built on frame count, not clip count.
The specification detail that decides everything
Frame rate, resolution, and whether annotation is frame-by-frame. These three set the order of magnitude of the dataset.
This is the line item that most often separates a dataset that works from one that gets re-ordered. It belongs in the specification before production starts, not in the delivery review.
How we source it
We do not hold inventory in this category. A requirement comes in, we match it to a producer who can meet the specification, and we agree the quality bar with both sides before recording begins. A pilot batch goes first so misalignments surface early.
Scenarios this covers
Buyers rarely ask for a category by name. They describe a situation, and the situation is what tells us whether this category is the right one. If yours is in the list, name it in the brief and it goes straight into the specification. If it is not, describe it anyway, because it usually means the requirement sits in a neighbouring category.
What good Video Data looks like
Video data is judged on frame rate, resolution and annotation granularity, which together set the order of magnitude of the work.
-
Frame rate and resolution stated
These determine what actions are even visible, and a corpus at a low frame rate cannot train a model for fine motion.
-
Annotation granularity stated
Clip-level labels, per-frame boxes, or temporal boundaries, because the three differ in cost by multiples.
-
Temporal boundaries with a consistency rule
Where events begin and end is a judgement, and the rule for it has to be in the guideline rather than left to the annotator.
-
Track continuity through occlusion
Whether an object that disappears and returns keeps the same identifier, because that decision changes what the data can train.
-
Clip provenance and duration distribution
Where the footage came from and the spread of clip lengths, because a corpus of uniformly short clips does not represent longer events.
Public datasets in this category
These are openly documented resources, not products we sell. Each one covers part of the category, and knowing where each stops is usually what decides whether a project needs a collection of its own.
-
Kinetics
Hundreds of thousands of short YouTube clips across several hundred human action classes. It is the standard action-recognition benchmark, and it stores links rather than video because of platform terms.
-
Something-Something
Short clips of object interactions with fine-grained action labels. The labels describe the physical interaction, which is why it is used for reasoning work rather than for recognition alone.
-
HowTo100M
Instructional videos with automatically transcribed narration, over a hundred million clip-text pairs. The scale is the contribution and the automatic narration is noisy.
-
ActivityNet
Untrimmed videos with temporal action boundaries and captions. The temporal annotation is the valuable part, since most corpora label only the clip.
Questions to ask a supplier
Questions that establish the annotation unit and whether the schedule reflects it.
- What is the frame rate and resolution, and is it constant across the corpus?
- Is the annotation clip-level, per-frame, or temporal boundaries?
- What is the rule for where an event begins and ends?
- How are objects tracked through occlusion?
- What is the clip duration distribution, and can you deliver it?
The vague answer in this category concerns what is being delivered. A supplier who says the corpus contains video without saying whether the video itself is included or only references has described two products with different lifetimes. Links expire when content is removed, and a corpus of links is a corpus that shrinks.
A verifiable answer states whether the files are video or references, the resolution and frame rate as delivered, the temporal annotation granularity, and the share of clips that could not be retrieved. Ask for the retrieval rate as measured at delivery rather than as intended at collection.
Technical attributes
The items below are the ones a specification can hold a producer to, and the ones that decide whether a delivery is usable for training rather than only present.
| Frame rate | Stated, constant across the corpus |
|---|---|
| Resolution | Stated per source |
| Annotation granularity | Clip, frame, or temporal boundary |
| Temporal rule | Documented for event boundaries |
| Track continuity | Policy for occlusion stated |
| Delivery format | Video with annotations in a named format and a per-clip manifest |
The temporal boundary column decides what the corpus can train, and it is the one most often left as a single label per clip. A clip labelled with one action usually contains several, and without boundaries the model learns to associate the whole clip with one label. The resulting confusion is between actions that co-occur.
The values come from the task. Action recognition needs clip-level labels, temporal detection needs boundaries, and video-language work needs aligned descriptions. The three need different annotation and different clip lengths, so the task has to be named before the specification is written.
A mistake here is discovered as confusion between classes. A corpus without temporal boundaries produces a model that cannot separate actions that appear together, and the error looks like a class-definition problem rather than an annotation problem. Fixing it means re-annotating the clips.
How Video Data is collected
The schedule is built on frames, because that is what the annotation cost follows.
- Task definition and annotation unit. The annotation unit is fixed from the task before any footage is collected, because it determines the cost and the schedule.
- Footage selection or capture. Footage is selected for the actions and conditions the model will meet, with resolution and frame rate set by what has to be visible.
- Guideline with temporal rules. The rules for event boundaries and occlusion are written and tested on a pilot batch, because those are where annotators diverge most.
- Annotation with consistency checks. Annotation runs with a defined proportion reviewed, and agreement is measured on the temporal boundaries specifically.
- Manifest and delivery. Per-clip metadata including duration, source, resolution and frame rate is delivered with the annotations.
After acquisition, the clips are checked for retrievability, temporal boundaries are annotated where required, and the files are packaged with per-clip metadata covering the resolution, the frame rate and the source. The retrieval check is what turns a link corpus into a delivery with a known content rate.
The pilot for video is a small set of clips checked for retrievability and annotated with temporal boundaries, run before the full acquisition. Its purpose is to measure the retrieval rate and to test whether the boundary rules hold on clips that contain several actions. Both are decided on the sample rather than on the finished delivery.
Acceptance is checked on the retrieval rate as measured at delivery and on the temporal annotation against the stated granularity. The specification should say whether the delivery is video files or references before acquisition starts, because the two have different lifetimes. Where the pilot shows the retrieval rate is low, the source is changed rather than the manifest adjusted.
How a delivery goes wrong
The defects below are the ones that recur across this category. Each is decided at collection or annotation time, which is why none of them can be repaired in delivery review.
-
Temporal boundaries are coarse
A clip labelled with one action often contains several. Without temporal boundaries the model learns to associate the whole clip with one label, and the error shows up as confusion between co-occurring actions.
-
Links expire
Corpora distributed as URLs rather than video break as content is removed. A delivery has to state whether the video itself is included or only the references, because the two have different lifetimes.
-
Frame rate and resolution were downsampled silently
Downsampling reduces storage and destroys motion information that some tasks depend on. The delivered rate belongs in the manifest, since it cannot be recovered.
Why this matters to buyers
Action recognition and tracking are core capabilities in video products, and the training material is expensive enough that corpora are reused across projects. What makes one reusable is documentation of the annotation unit and the temporal rules.
The second driver is cost control. Because video annotation scales with frames rather than clips, a specification that fixes the granularity after quoting has changed the project's cost by a multiple without changing its description.
A model trained on clips without temporal boundaries fails on the actions that co-occur. The failures look like taxonomy problems and are actually annotation problems, so the team revises the class list instead of the corpus. That loop can run several times before the real cause is found.
The second cost is the shrinking corpus. Where the delivery is references rather than video, content disappears over time and the corpus cannot be reproduced. Teams training a second model a year later find they are working with less data than the first time, and the comparison between the two is no longer valid.
A video corpus with boundaries annotated gives a model that can separate actions which appear together, which is the failure this category otherwise produces. The team avoids a re-annotation pass, and the class list does not have to be rewritten around the annotation.
The second gain is that the delivery is reproducible. Where files rather than links are delivered and the source is recorded, a second model trained a year later works on the same content, so the two can be compared. That comparability is what makes a video programme measurable.
Specification at a glance
| Category | Video Data |
|---|---|
| Group | Non-speech data |
| Critical spec | Frame rate, resolution, and whether annotation is frame-by-frame. These three set the order of magnitude of the dataset. |
| Included by default | Pilot batch, metadata schema, source and consent documentation |
| Pricing | Quoted from specification. No published rates |
Related categories
-
LLM Training Data
Text corpora for LLM pretraining and fine-tuning: instruction data, preference data, and multi-turn dialogue data.
-
Computer Vision Data
Annotated image and video data, with annotation types including classification, bounding boxes, segmentation masks, and keypoints.
-
NLP Data
Annotated text for NLP tasks: named entities, dependency syntax, sentiment polarity, and relation extraction.
-
Multilingual Language Data
Parallel or non-parallel text corpora spanning multiple languages, used for multilingual models and low-resource language research.
-
Translation Data
Parallel corpora pairing source and target language, used for machine translation training and evaluation.
-
Dictionary Data
Structured dictionary entries with definitions, parts of speech, pronunciation, and example sentences, used for dictionary products and lexical understanding tasks.
Questions about Video Data
Why build the schedule on frames rather than clips?
Because the annotation time follows frames. Ten thousand clips of three seconds and ten thousand clips of three minutes are the same number of clips and a hundredfold difference in work.
How is an event boundary defined?
By a written rule tested on a pilot batch. Where an action starts is a judgement, and without a rule the boundaries vary by annotator and the temporal labels become inconsistent.
What happens to an object that is occluded?
Per a stated policy. The common choices are to keep the track alive through the occlusion or to terminate and restart it, and both are workable. What matters is that the policy is documented, because a model trained on undocumented track breaks learns them as real.
Can we downsample the video to reduce cost?
Only if the actions remain visible. Frame rate determines what motion is observable, and a corpus downsampled below the rate the task needs cannot train it regardless of how many hours it contains.
Do you have Video Data available now?
No. We do not hold inventory. Everything is produced against a specification. That means the timeline starts when the spec is agreed rather than immediately, and it also means the data matches your requirement instead of approximately matching something already built.
Can this be combined with other categories in one delivery?
Yes, and it is common. A single specification can cover several categories across several languages, with one pilot batch and one quality bar. It usually reduces total cost because contributor recruitment and project setup are shared.
What does the consent documentation cover?
Signed contributor consent stating the intended use, the collection methodology, and the annotation guideline the data was produced under. For projects involving EU data subjects the chain is built to satisfy GDPR, including the transfer mechanism. The terms for moving the data across a border.
Request Video Data
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.