What is Embodied AI Data?

What separates embodied data from ordinary video or text is the action record: a sample is a loop of perception, action and consequence, not a description. That loop can come from three sources (a real robot under teleoperation, a person (egocentric video or motion capture), or a simulator), and the three are not interchangeable.

Each source buys something different. Real-robot data carries true contact physics and is the most expensive per hour. Human data is cheap by comparison and scales to thousands of hours, but contains no robot actions. It teaches perception and motion priors, not control. Simulation is nearly free per sample and can generate failures on demand, but it is only as good as its physics. A serious program mixes all three, and the mix is a decision made before collection starts, not after.

One trap to avoid: hours are not comparable across sources. An hour of teleoperation data, an hour of egocentric video and an hour of simulated episodes are different products at different costs with different uses, so a specification written in total hours hides the decision that matters, which is why briefs in this category are written as a task list and a sensor configuration. Nothing here is held in stock; projects are collected, processed or licensed to order and built around the specification.

Kinds of embodied data

The first form is the demonstration: a human drives or guides the robot, and the episode records observation, action and outcome together. This is the only form that pairs true robot actions with real contact physics, which is why it is the most expensive per hour and the form policies are usually fine-tuned on.

The second form is human observation without a robot: egocentric video, motion capture or hand tracking. There are no robot actions in it, so it cannot teach control directly, but it scales to thousands of hours at a fraction of the cost and carries the perceptual and motion priors that make the expensive robot data go further.

The third form is simulated rollout, where the loop is generated rather than recorded. It is nearly free per sample, can produce failure cases on demand, and is bounded by the accuracy of its physics. A fourth form, less often used for training, is operational logs from robots already deployed: real outcomes at scale, but with actions chosen by the system being improved rather than by an expert.

How a brief is written

A brief in this category is a task list and a sensor configuration, not an hour count. The task list names each behavior, the objects involved and the conditions it must be demonstrated under. The sensor configuration names every stream, its rate and where it is mounted. An hour figure on its own cannot be quoted, because hours from different sources are different products.

Specify the resets. Throughput is set as much by how quickly the scene can be returned to its start state as by how fast the task runs, and a task whose reset takes longer than its demonstration costs more per usable episode than it appears to. Say also whether reset footage is trimmed from delivery, because a policy trained on it learns to copy the operator rather than the task.

Specify coverage and the held-out set: how many scenes, objects, lighting conditions and operators are involved, and which of them are reserved for evaluation. Embodied models fail on the arrangement they have never seen, so the held-out portion of the specification is what predicts how the data will perform in the field. Fix this before collection starts, because coverage cannot be added afterwards.