What is Robot Data Augmentation?

At the image level the techniques are the familiar ones (crops, color jitter, texture and lighting randomization), and they buy visual robustness cheaply. At the episode level the interesting methods are geometric: MimicGen takes a source demonstration and re-renders it onto new object poses, new receptacles and even a different robot arm, turning tens of human demonstrations into thousands of synthetic episodes.

Relabeling is the other lever. A trajectory that failed to reach its intended goal can be relabeled with the goal it did reach, and a demonstration recorded without language can be given instructions after the fact. Both multiply the useful content of data that was already paid for.

What augmentation cannot do is invent physics. No transformation of a tabletop episode produces the contact dynamics of cloth, and image-level processing cannot create a sensor modality that was not recorded. That is the honest test of whether to buy augmentation or more data: if the missing variation is geometric or visual, augmentation is cheap and effective; if it is contact or sensing, it is neither. Augmentation is normally scoped as part of a collection or conversion engagement rather than bought on its own, because the useful question is what your existing data lacks.

Types of augmentation

The cheapest kind works on single frames. Crops, color jitter, blur, noise and randomization of lighting and surface texture are applied to the images in an episode while the action labels stay untouched. The motion did not change; only what the cameras saw did. This buys robustness to visual variation at very little cost, and it is the kind most pipelines apply by default.

The second kind is geometric. A recorded demonstration is replayed through a new scene: the same action sequence against new object poses, a new receptacle, or even a different robot arm. The trajectory is re-rendered and the labels follow the motion, which is how a small human collection is multiplied into a much larger set. This kind changes where things are, not how they look.

The third kind transforms no pixels at all. It relabels. A trajectory that failed to reach its intended goal is re-annotated with the goal it did reach, and a demonstration recorded without language can be given an instruction afterwards. Nothing new is recorded and the motion is the same, but each episode now carries more usable supervision. It is often the cheapest useful augmentation available.

What to specify when buying

Augmentation is rarely bought on its own; it is scoped inside a collection or conversion engagement. So the specification starts with a diagnosis, not a technique: which variation is missing. If the policy fails when an object sits somewhere new, that is a geometric problem. If it fails when the lighting changes, it is a visual one. The brief names the failure; the supplier proposes the method.

The second item is the target rig. Augmented episodes have to be valid for the robot the policy will run on, so the brief names the arm, the camera positions, the control rate and the file format. An episode re-rendered onto the wrong embodiment is not free data; it is noise that costs money to generate and then has to be filtered out again.

The third item is validation, and it has two parts. Splits must be drawn by source episode, not by augmented sample, or near-duplicate versions of one demonstration will land in both training and test. And the external measure is a comparison: train one policy on the original data and one on the augmented set, then evaluate both on real hardware.