Embodied AI data
Data for systems that act in the physical world rather than only describe it. The hard part is not collecting it. It is collecting it in a way that generalises beyond the room it was recorded in.
What counts as embodied AI data
Embodied AI covers robots, autonomous vehicles, drones, and software agents that control something in the physical world. What separates its data from ordinary training data is that every sample carries a record of action and consequence: what the system perceived, what it did, and what happened next.
A caption on an image says what is in the picture. An embodied sample says where the arm was, how it moved, whether it made contact, and whether the grasp held. That extra structure is what makes the data expensive and what makes it useful.
The same request means five different things depending on where you are. Most buyers arrive wanting one of the five groups below: a benchmark to compare against, a way to get the data captured, the manipulation task they actually run, the environment the robot works in, or the terms. Start from the one you recognise.
Datasets and benchmarks
Open corpora that teams train on or evaluate against, and the format questions that decide whether you can use them at all.
-
DROID Dataset
A large in-the-wild robot manipulation dataset collected by a multi-lab consortium, notable for scene diversity rather than task repetition.
-
Open X-Embodiment
A pooled collection of robot datasets spanning 22 different robot embodiments, built to test whether data from one robot helps another.
-
LeRobot
An open-source library and dataset format for robot learning, which fixes how episodes, video and state are stored so datasets can be shared.
-
LIBERO
A simulation benchmark for robot manipulation built to measure knowledge transfer, with four task suites that isolate different kinds of generalization.
-
AgiBot World
A large real-world robot dataset collected on a standardized humanoid platform, notable for being recorded at production scale by one organization.
Scenario tags
How the data is produced
The capture and post-processing routes. This is where the cost sits, and where two suppliers quoting the same hour count can be quoting two different products.
-
Teleoperation
A human drives the robot directly, and the robot's motion is recorded as a training demonstration.
-
Egocentric Video Data
First-person footage, usually from a head-mounted camera, showing hands and objects from the wearer's own viewpoint.
-
Motion Capture Data
Human body motion recorded as joint trajectories, used for animation and as a reference for robots that imitate people.
-
Sim-to-Real Transfer
Training a policy in simulation and getting it to work on physical hardware, across the gap between the two.
-
Robot Data Annotation
Attaching labels to robot episodes, recording what the task was, whether it succeeded, and when key events happened.
-
Robot Data Augmentation
Generating additional training episodes from existing ones by transforming images, goals or scene configurations.
Scenario tags
Manipulation and grasping
What the hands and grippers are doing. Buyers who arrive with a task name get a scoped quote; buyers who arrive with "robot data" get a question back.
-
Robot Manipulation Data
Episodes of a robot arm moving objects, recorded together with the commands that produced the motion.
-
Grasping Data
Images, depth maps or point clouds of objects, annotated with the gripper poses that would pick each one up.
-
Dexterous Hand Data
Demonstrations for multi-fingered robot hands, including in-hand manipulation that a two-finger gripper cannot do.
-
Robot Gripper Data
Episodes and signals tied to the end effector, including width, force, vacuum pressure and whether the grasp held.
-
Tactile Sensor Data
Contact measurements from a robot's fingertips or skin, covering pressure, force and, in optical sensors, images of the contact itself.
-
Vision-Language-Action Model
A model that takes camera images and a written or spoken instruction and outputs robot actions directly.
Scenario tags
Robots, environments and learning
Where the robot has to move, and the learning setups that turn a task list into a policy. Navigation data is specified by environment, not by task.
-
Humanoid Robot Data
Whole-body demonstrations and sensor logs from robots with legs and arms, where balance is part of every task.
-
Mobile Robot Navigation Data
Sensor logs of a robot moving through space, with the poses, maps and trajectories needed to train or evaluate navigation.
-
Warehouse Robotics Data
Sensor and telemetry data from robots working in fulfillment centers, covering picking, moving totes and building pallets.
-
Imitation Learning
Teaching a policy by showing it demonstrations rather than by scoring its attempts, which is why demonstration data is the input it needs.
-
World Model
A learned model that predicts how a scene will change in response to actions, used both for planning and for generating training data.
Scenario tags
Buying, specifications and compliance
What has to be agreed before capture starts. Every item on this list is something that cannot be recovered from the recordings afterwards.
-
Embodied AI data collection
How a collection project is scoped, what decides the cost, and how long an engagement runs.
-
Robot data collection services
What a collection service actually provides, and what to check before you sign.
-
Physical AI data
The wider term, and which parts of it are collectable to order.
-
Choosing a data provider
What separates a producer that can hold a specification from one that cannot.
-
Embodied AI Data
The catalogue entry: what this category covers and how it is specified.
-
Compliance, consent and licensing
How consent, bystander handling and site permission are documented.
Scenario tags
Why it is expensive
Cost is driven by time-on-task rather than by volume. An hour of ordinary speech data is one hour of a speaker talking. An hour of teleoperation is one hour of a trained operator standing at a rig, plus the rig, plus the objects being manipulated, plus the resets between attempts.
Two consequences follow. First, the same scene shot from a different angle is cheap; the same scene with a different task is not. Second, projects should be scoped around the number of distinct tasks and object types, not around hours, because that is what actually determines how long the collection takes.
The generalisation problem
A dataset recorded in one kitchen produces a model that works in that kitchen. The variability that matters is not the number of repetitions but the spread of conditions: different lighting, different surface materials, different object shapes within the same category, different clutter levels, different camera mounting positions.
This is why embodied collection is usually specified as a matrix rather than a total. Thirty hours spread across six environments and forty object categories teaches more than thirty hours in one room, even though the file count is identical.
What to specify
These are the fields that cannot be recovered from the recordings afterwards. Settle them in the specification, not in the delivery conversation.
- Task list:the concrete actions, written as verb plus object, not as a category name.
- Environment spread:how many distinct locations, and what varies between them.
- Object inventory:the physical items needed, including how many of each and whether they must be sourced by the producer.
- Sensor configuration:camera model, mounting, frame rate, whether depth is captured, whether force or tactile sensing is included.
- Robot platform:the exact model, since joint conventions and action spaces differ between them.
- Calibration record:intrinsic and extrinsic camera parameters, without which multi-view data cannot be fused.
- Success criteria:what counts as a completed attempt, and whether failed attempts are retained.
- Delivery format:RLDS, HDF5 episodes or ROS bags, and whether the action space is stored absolute or relative.
Questions we get asked
What is the difference between embodied AI data and physical AI data?
Physical AI is the wider term and covers anything that senses and acts in the physical world, including autonomous vehicles, drones and industrial inspection. Embodied AI is the part of it where the system has a body it has to control, so robot arms, mobile robots and humanoids. Every embodied AI dataset is physical AI data, but a road-going driving dataset is physical AI without being embodied.
Can you collect data on our robot platform?
Usually, provided the platform exposes its joint states and the teleoperation route in. What we need from you is the exact model, the firmware version, and whether the action space is joint positions, joint velocities or end-effector poses. Where the platform is unusual or the SDK does not expose state at the rate the task needs, we say so before quoting rather than after.
How many hours do we need?
Hours is the wrong unit for embodied work and it is the unit most quotes are written in. What decides whether a policy transfers is the number of distinct tasks, environments and object types, not the total duration. Thirty hours spread across six rooms and forty object categories teaches more than thirty hours in one room. We scope against a matrix, then convert it to hours.
Is simulation cheaper than real capture?
Per episode, yes, and by a wide margin. Per usable episode, sometimes not. Simulated data is only worth producing where the domain gap can be closed, which usually means randomising the specific things that differ between the simulator and your deployment site. Where the task depends on contact, deformation or material behaviour, real capture is normally the cheaper route once the retraining cycles are counted.
Can you match the format of a published benchmark?
Yes, and it is worth doing early. RLDS, HDF5 episode files and ROS bags are the three formats most projects settle on, and each one fixes things like whether the action space is stored absolute or relative. Converting after collection is possible but loses information that was never recorded, so the format belongs in the specification rather than in the delivery conversation.
What happens to people who appear in the recordings?
Egocentric and multi-view capture routinely records people who never agreed to it. The controls are to record in controlled spaces where possible, to apply a defined blurring or exclusion rule to bystanders before delivery, and to document what that rule was. Where someone appears deliberately, ordinary consent requirements apply exactly as they would for voice data.
Submit a sourcing request
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.