Inter-Annotator Agreement
Short answer. Inter-annotator agreement measures whether two people labeling the same items produce the same labels. Raw percent agreement is misleading because some of it happens by chance: with three balanced classes, two people guessing independently agree a third of the time. Cohen's kappa corrects for that by subtracting the chance agreement and rescaling, so 0 means chance and 1 means perfect. The number is the cheapest evidence that a guideline is being applied the same way by everyone, which is why it belongs in a delivery acceptance test rather than in a methods appendix.
How to do it
Why percent agreement is not enough
Two annotators working on a three-class task with balanced classes agree by chance about a third of the time. So a reported agreement of 0.85 sounds strong until you notice that a third of it was free. The same 0.85 on a two-class task with one very rare class is a much weaker result, because the chance floor is different.
Kappa removes the floor by computing the agreement expected from the marginal label frequencies and rescaling the observed agreement against it. The formula is observed minus expected, divided by one minus expected. It is not a correction for error, it is a change of scale, and it is why kappa values are lower than percent agreement on the same data.
- po = fraction of items where the two annotators agreed
- pe = chance agreement from the label marginals
- kappa = (po - pe) / (1 - pe)
Reading a kappa value
The bands quoted in textbooks are a rough guide and they were written for a different purpose: above 0.8 is usually described as strong, 0.6 to 0.8 as moderate, and below 0.4 as weak. What matters more is the comparison within a project. The same task with the same guideline typically lands between 0.6 and 0.8, and a value far above that usually means the task is easy rather than that the annotators are good.
A low value has two possible causes and they need different fixes. Either the guideline is ambiguous, in which case the disagreements cluster on specific items and reading them tells you what to clarify, or the task is genuinely subjective, in which case no guideline will fix it and the label scheme needs to change. Look at the disagreements before deciding which one you have.
Using agreement as an acceptance gate
The useful place for agreement is before a batch starts, not after it ends. Have two annotators label a few hundred items independently, compute agreement per class, and resolve the disagreements with the person who wrote the guideline. That pilot produces a clarified guideline and a number, and the number is what the rest of the batch can be held to.
Report it per class rather than overall. The hard classes are usually the rare ones and the ones with unclear boundaries, and an overall figure averages them away. A delivery with an overall agreement of 0.8 where one class sits at 0.45 is a delivery with a problem in that class.
Code
Compute percent agreement and Cohen's kappa for two annotators on twenty items, show the confusion matrix, and watch both numbers move when the disagreements are resolved.
import numpy as np
LABELS = ["positive", "negative", "neutral"]
# 20 reviews, with the label each of two annotators assigned
A = ["positive", "negative", "neutral", "positive", "negative",
"negative", "neutral", "positive", "positive", "negative",
"neutral", "negative", "positive", "neutral", "negative",
"positive", "negative", "neutral", "positive", "negative"]
B = ["positive", "negative", "neutral", "positive", "neutral",
"negative", "neutral", "positive", "negative", "negative",
"neutral", "negative", "positive", "neutral", "negative",
"positive", "negative", "neutral", "positive", "neutral"]
idx = {l: i for i, l in enumerate(LABELS)}
ia = np.array([idx[x] for x in A])
ib = np.array([idx[x] for x in B])
n = len(A)
agree = int((ia == ib).sum())
po = agree / n
print("items %d, agreed %d, observed agreement %.3f" % (n, agree, po))
print("disagreed on items: %s"
% [i + 1 for i in range(n) if ia[i] != ib[i]])
print()
M = np.zeros((len(LABELS), len(LABELS)), dtype=int)
for x, y in zip(ia, ib):
M[x, y] += 1
print("confusion matrix, rows = annotator A, cols = annotator B")
print(" " + "".join("%10s" % l for l in LABELS))
for i, l in enumerate(LABELS):
print("%-12s" % l + "".join("%10d" % v for v in M[i]))
pe = float((M.sum(axis=0) * M.sum(axis=1)).sum()) / (n * n)
kappa = (po - pe) / (1.0 - pe)
print()
print("chance agreement pe = %.3f" % pe)
print("Cohen's kappa k = %.3f" % kappa)
# resolve the disagreements on the two rare classes and watch kappa move
B2 = list(B)
B2[4] = A[4]
B2[8] = A[8]
B2[19] = A[19]
ib2 = np.array([idx[x] for x in B2])
po2 = float((ia == ib2).mean())
M2 = np.zeros((len(LABELS), len(LABELS)), dtype=int)
for x, y in zip(ia, ib2):
M2[x, y] += 1
pe2 = float((M2.sum(axis=0) * M2.sum(axis=1)).sum()) / (n * n)
k2 = (po2 - pe2) / (1.0 - pe2)
print()
print("after an adjudicator resolves all three disagreements:")
print(" observed agreement %.3f -> %.3f (+%.3f)" % (po, po2, po2 - po))
print(" Cohen's kappa %.3f -> %.3f (+%.3f)" % (kappa, k2, k2 - kappa))- pip install numpy.
- Expected: 17 of 20 items agree, so observed agreement is 0.850. The chance agreement is 0.333, and Cohen's kappa comes out 0.775. Resolving the three disagreements moves observed agreement by 0.150 and kappa by 0.225.
- That gap is the point. Percent agreement and kappa are on different scales, and kappa moves further for the same number of fixes, so a target written as 0.9 agreement is not the same requirement as a target written as kappa 0.9. Write down which one the acceptance criteria mean.
- The confusion matrix shows where the disagreement sits: annotator A used positive seven times and annotator B used it six, and the two neutral cells in the negative row are the cases where one saw a mixed review and the other saw a negative one. Reading those two items is what produces the guideline clarification.
Where this goes wrong
Reporting percent agreement and calling it agreement
On a three-class task a third of the agreement is free, so 0.85 means the annotators beat chance by about half. Reporting the raw number invites the reader to treat it as a quality figure. Report kappa, or report both and say which is which.
Computing agreement on a sample chosen by the annotators
If the annotators pick the items to double-label, they pick the easy ones and the number is meaningless. The sample has to be drawn by the person running the project, at random, before annotation starts. This is the single most common way an agreement figure becomes decoration.
Averaging over classes when one class is broken
A delivery with an overall kappa of 0.8 and one class at 0.45 passes an overall gate while the class that matters sits below any usable threshold. Compute agreement per class, and if the product depends on a specific class, write that class into the acceptance criteria on its own.
Treating a low kappa as an annotator problem
A low value usually means the guideline does not decide the cases the annotators are meeting, not that the annotators are careless. Read the disagreements before replacing anyone: if they cluster on one kind of item, the guideline needs a rule for that kind, and the same annotators will agree once it has one. Replacing people without fixing the guideline reproduces the same number with a new crew.
When to buy the data instead of building the pipeline
Compute the agreement yourself, because the sample has to be yours and the disagreements are the deliverable you actually want to read. Buy the annotation when the class list is long, the volume is beyond your team, or the domain needs knowledge you do not have. Ask for a two-annotator pilot with per-class agreement reported before the main batch starts, and treat the pilot as the acceptance test rather than a formality. We run a two-annotator pilot on a random sample with per-class agreement reported, and the resolved disagreements feed back into the guideline before the batch is produced.
Need data for Inter-Annotator Agreement?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.