What is Annotation?

Annotation quality comes down to two things: how detailed the guideline is, and whether anyone cross-checks the work.

The industry measure is inter-annotator agreement: have different people annotate the same batch and see how far apart they land. The wider the gap, the vaguer the guideline.

How annotation is scoped

Annotation is scoped by how fine the label is. At the coarsest, a whole recording carries one label: this call is a billing complaint. At the middle, segments carry labels: this stretch is speaker A, this stretch is background music. At the finest, individual words, frames or pixels are labelled. Cost rises steeply as the unit gets smaller, and most quote confusion comes from two parties meaning different units.

The next distinction is the task itself. Transcription writes out what was said. Classification picks one label from a fixed list. Sequence labelling assigns a label to every unit in order, as diarization and phonetic annotation do. Extraction pulls specific items out of free speech, such as a name or address. Verdicts are their own category: whether a call was resolved, whether a recording sounds natural.

Finally, the label set itself. A closed set, meaning fixed categories agreed in advance, is easier to produce and easier to measure agreement on. An open set, where annotators write free text, is more expressive and much harder to check, because two people can write different sentences that mean the same thing. Single-pass annotation uses one person per item; a multi-pass process adds an adjudicator.

Where it shows up in a quote

Annotation is priced by the unit of work, not by the hour of audio. A quote is per audio hour for transcription, per segment for speaker labelling, per frame for masks, per item for classification. The unit is what to compare between suppliers. Two quotes that both read per hour are not comparable if one includes time alignment and quality checks and the other does not.

The guideline and the label taxonomy are deliverables, and they should exist before production starts. A serious process annotates a small pilot batch with several people, measures where they disagree, rewrites the ambiguous parts of the guideline, and only then scales up. A supplier who begins without that step discovers the ambiguity in the middle of the batch, when work already done has to be redone.

What to require in the contract: a stated proportion of double-annotated work, a stated agreement figure, an adjudication step for disagreements, and terms for rework if the delivery fails the acceptance sample. One more item is easy to forget. Someone on the buyer side has to answer the questions annotators raise. If no one does, the annotators decide for themselves, and those decisions become your conventions.