What is Robot Manipulation Data?

A manipulation episode is a time series, not a clip: camera frames, joint positions, gripper state and the action command, all aligned on one clock. The control rate is the first number that matters. ALOHA-style bimanual rigs record at 50 Hz, while a large multi-lab set like BridgeData V2 runs at 5 Hz, and a policy trained at one rate does not transfer cleanly to the other.

The common formats are HDF5 (one group per episode, the layout ALOHA and ACT use), RLDS, which is how Open X-Embodiment is distributed, and the LeRobot format, which keeps low-dimensional state and action in Parquet files and camera streams as MP4. How many demonstrations a task needs is a scoping question: the ACT work found about 50 per task sufficient for fine bimanual skills, while RT-1 collected over 130,000 episodes across more than 700 tasks: roughly 150 to 200 per task.

What makes this data hard is that demonstrations are only as useful as their coverage. Fifty episodes of the same grasp teach one grasp; the value is in object variety, lighting, camera placement and the failure cases most producers discard. Sets of this kind are built to order. The task list, platform and sensor configuration are specified first, and the episode count follows from them.

Types of manipulation tasks

Manipulation tasks sort into families by what the hand has to control. Pick-and-place is the simple end: the object is rigid, the target is a surface, and success tolerates a few centimeters of error. Insertion and assembly are the hard end (a peg into a hole, a connector into a socket) decided in the last millimeter of motion. A dataset covering one family says little about the others.

A second split is by what the object does under the hand. Rigid objects keep their shape. Articulated ones (drawers, doors, hinged lids) carry joints the robot has to discover. Deformable material (cloth, cable, food) changes shape while it is handled, and simulation models it worst. Each family changes the sensing that matters: rigid pick-and-place runs on camera and joint state alone; cloth usually cannot.

Platform configuration is the third axis. A single arm bolted to a table is the simplest stream set. A bimanual rig doubles the arms and adds coordination between them. A mobile manipulator adds base pose and localization on top. The further a rig sits from the platform your policy will run on, the more the data is a reference rather than a substitute.

What to specify when buying

The brief for manipulation data is a task list, not an hour count. Each task needs its objects, its start states and its success criterion written down. The success criterion is where briefs go vague. "Did the grasp succeed?" is obvious until the object slips two seconds after release. A hold period and a placement tolerance turn that argument into a rule.

Platform and sensor configuration set both the price and how reusable the result is. The brief should name the arm, the gripper, the camera count and placement, and the control rate. Control rate is part of the specification, not a detail the supplier can pick: a stack built around one rate has to resample data recorded at another, and every resample risks losing the alignment between frames and joint states.

Coverage is the last item and the easiest to leave implicit. Episode counts per task matter less than how objects, lighting and camera placement vary across them; a large set from one room still describes one room. The episodes producers are tempted to drop (the slip, the knock-over, the retry) teach a policy what going wrong looks like. Say whether they are kept.