What is Inter-Annotator Agreement?

High agreement means the guideline is clear and the task is executable. Low agreement is usually not an annotator problem; it means the guideline itself is ambiguous.

You can ask a supplier for this number. The ones who can produce it usually run a formal annotation process.

How agreement is measured

Agreement is reported as a number, but the number can be computed in different ways. The simplest is the share of items on which annotators landed on the same label. That is easy to read and misleading on lopsided tasks: if nine clips in ten contain no target event, an annotator who marks nothing at all agrees with almost everyone.

The standard correction is a chance-adjusted measure, which subtracts the agreement you would get from guessing. Cohen's kappa covers two annotators; Fleiss' kappa and Krippendorff's alpha extend to more, and alpha also tolerates missing labels. Which one was used matters less than saying which one it was, because the numbers are not comparable across measures, and a figure quoted without its measure is hard to use.

For labels that are positions rather than categories, agreement is measured as distance: how far apart two annotators placed a boundary, and whether that gap falls inside a tolerance. Prosody, diarization and event timestamps all work this way. The tolerance has to be agreed before the measurement means anything, and it should be quoted with the number, because a loose tolerance and a strict match are different results.

What to specify when buying

Ask the supplier for the agreement number and the measure behind it. A supplier who can produce it usually runs a formal process with a written guideline; one who cannot is describing a process rather than a measured result. Ask what it was measured on, too: a pilot batch is more informative than the full run, because a low number there can still change the guideline.

The number is a diagnostic, not a grade. Low agreement on a subjective task is expected and can be managed with tolerances; low agreement on a task that should be mechanical, such as whether a clip contains speech, points at an ambiguous guideline. The right response is to fix the guideline and re-run the batch, not to replace the annotators.

Agreement is also worth asking for per label, not only per batch. A batch can show high agreement overall while the one field that matters to the model, the success verdict or the emotion tag, is the field annotators cannot agree on. That is the number the buyer should be reading, and it is worth writing an acceptance threshold into the brief before the work starts.