What is Robot Data Annotation?

Robot annotation is mostly episode-level and temporal rather than spatial. The typical labels are a natural-language instruction for the episode, a success or failure verdict, contact events with timestamps, and sometimes object segmentation masks for key frames. The tooling is general-purpose (CVAT and Label Studio both handle video and time series), so the cost sits in the guideline, not the software.

The success label is where annotators disagree. "Did the grasp succeed?" has an obvious answer only until the object is dropped two seconds after release, or placed slightly off target, or handed over in the wrong orientation. A guideline that defines a hold period and a placement tolerance converts those arguments into a rule; without one, the same episode gets different labels from different annotators.

Pricing follows label depth and is quoted per episode rather than per hour: templated language labels and success verdicts commonly fall in the range of a few cents to a few tens of cents per episode, while per-frame segmentation masks are priced per image at the dollar level. Where success is already implicit in the teleoperation flow, no post-hoc verdict is needed at all. Annotating existing footage is one of the services most often commissioned on its own, and the deliverable that decides quality is the written convention.

Kinds of annotation

Episode-level labels attach once to a whole episode: the instruction in natural language, a success or failure verdict, and administrative fields such as the operator, the rig and the objects present. They are the cheapest labels to produce and the ones most projects actually need, because a policy trained on demonstrations has to learn what each demonstration was for.

Temporal labels attach to moments: first contact, grasp, release, drop and the boundaries between sub-tasks. They are what let a long episode be cut into stages, and what let a policy be evaluated at the second that decided the outcome. The guideline has to define each event precisely, down to which frame counts as the start, or two annotators will place the same event seconds apart.

Spatial labels attach to individual frames: bounding boxes, segmentation masks, keypoints. They are the most expensive by a wide margin, so they are usually applied to sampled frames or to a named object rather than to every frame. A quote should say which of the three levels it covers, because the same footage annotated at episode level and at pixel level are two different products.

What to specify when buying

Ask to see the written guideline before accepting a quote. It is where the cost sits, and it has to settle what annotators would otherwise argue about: what counts as a successful grasp, how long an object must stay held before the episode counts as a success, how far off target a placement may be and which style the instructions follow. A supplier who cannot show a guideline cannot show consistent labels.

Specify the level and the sampling. If spatial labels are wanted, say whether they cover every frame, key frames only or a fixed sample, since per-image pricing makes that choice the main cost lever. If the target is a vision-language-action model, say how many independent instructions each episode should carry and in what style, because instruction phrasing is training input, not documentation.

Specify what happens to failures. Episodes where the robot dropped the object, missed the target or needed a correction are the ones a policy learns most from, and the ones default collection practice quietly discards. Say whether failures are kept, labeled and delivered, and fix how labels are joined back to the data, by timestamp or frame index.