Cohen's Kappa
Short answer. Cohen's kappa measures how much two annotators agree beyond what chance would produce. It subtracts the agreement expected from the label distribution and rescales what is left, so a high raw agreement on a skewed task can still score near zero. That is the point of the statistic and also its trap: kappa depends on the label distribution, so it is not comparable across tasks with different distributions, and a bare kappa does not tell you whether the annotators were careful or lucky.
How to do it
What chance agreement means here
If one annotator labels ninety percent of items positive and the other does the same, two annotators guessing independently will agree on roughly eighty-two percent of items without reading any of them. Kappa subtracts that expected agreement and rescales the remainder, so the score describes only the agreement that the label distribution does not already explain.
The expected agreement is computed from the marginals: the probability that annotator A says positive times the probability that B says positive, plus the same for negative. It assumes the two annotators are independent, which is the assumption the whole statistic rests on and the one that breaks when both were trained on the same guideline and share the same misunderstanding.
- po: the share of items the two annotators labelled the same
- pe: the agreement expected if both labelled independently at their own rates
- kappa = (po - pe) / (1 - pe)
Why the same agreement can score 0.80 or 0.00
On a balanced two-class task with ninety percent agreement, kappa is 0.800. On a task where ninety percent of items are one class, the same ninety percent agreement gives 0.444, because chance alone would have produced eighty-two percent. The observed agreement is identical and the statistic differs by nearly a factor of two.
The extreme case is an annotator who uses one label for everything. That annotator agrees with a second annotator on every item the second one also called positive, and the score is 0.000, because the expected agreement is exactly the observed agreement. Kappa is not broken here; it is reporting that the labels carry no information about the annotator decisions.
What to report alongside it
Kappa cannot be read alone, so a report needs the label distribution, the raw agreement, and the confusion matrix. The matrix is what distinguishes two careful annotators on a skewed task from one annotator who stopped reading, and those two cases can produce the same kappa.
Kappa also does not measure correctness. Two annotators who share the same misreading of the guideline will agree closely and both be wrong, and the score will be high. Agreement is a property of the annotators against each other; correctness needs an audit against the guideline, which is a different measurement with its own sample size.
Code
Four cases with the same 100 items and two classes, showing that observed agreement and kappa can move in opposite directions.
# four cases, 100 items each, two classes
# each cell is (annotator A's label, annotator B's label)
CASES = {
"balanced, 90 percent agreement": (
[("pos", "pos")] * 45 + [("pos", "neg")] * 5 +
[("neg", "pos")] * 5 + [("neg", "neg")] * 45),
"skewed 90/10, same 90 percent agreement": (
[("pos", "pos")] * 85 + [("pos", "neg")] * 5 +
[("neg", "pos")] * 5 + [("neg", "neg")] * 5),
"one annotator labels everything pos": (
[("pos", "pos")] * 90 + [("pos", "neg")] * 10),
"opposite biases, 10 percent agreement": (
[("pos", "pos")] * 5 + [("pos", "neg")] * 90 +
[("neg", "neg")] * 5),
}
print("%-42s %6s %6s %7s" % ("case", "po", "pe", "kappa"))
for name, pairs in CASES.items():
n = len(pairs)
po = sum(1 for a, b in pairs if a == b) / n
a_pos = sum(1 for a, _ in pairs if a == "pos") / n
b_pos = sum(1 for _, b in pairs if b == "pos") / n
pe = a_pos * b_pos + (1 - a_pos) * (1 - b_pos)
k = (po - pe) / (1 - pe) if pe < 1 else 0.0
print("%-42s %6.3f %6.3f %7.3f" % (name, po, pe, k))
print()
print("the first two rows have the same observed agreement and kappa differs by")
print("almost a factor of two. the third row agrees on 90 percent of items and")
print("scores 0.000, because an annotator who uses one label for everything can be")
print("matched by guessing that label, so the agreement carries no information.")
print()
print("this is why kappa must be reported next to the label distribution, and why a")
print("confusion matrix belongs in every agreement report. a bare kappa of 0.44 can")
print("mean two careful annotators on a skewed task, or one annotator who stopped")
print("reading. the matrix tells them apart; the number does not.")- Standard library only. This computes kappa for two annotators and two classes. More than two annotators needs Fleiss kappa, which generalises the same expected-agreement idea.
- Expected: balanced gives po 0.900, pe 0.500 and kappa 0.800. Skewed 90/10 gives the same po 0.900 with pe 0.820 and kappa 0.444. The first two rows differ only in the label distribution.
- Expected: the third row, where one annotator uses a single label for all 100 items, gives po 0.900 and kappa exactly 0.000. The fourth row, where the two annotators have opposite biases, gives po 0.100 and kappa 0.006, which is what no agreement beyond chance looks like.
- Change the first case to 95 percent agreement and kappa rises to 0.900 while the fourth case stays near zero, which is the behaviour you want from the statistic.
Where this goes wrong
Comparing kappa across tasks with different label distributions
Kappa is not comparable between a balanced task and a skewed one, because the expected agreement term is different in each. A project that reports 0.6 on one dataset and 0.6 on another has not shown the annotation quality is the same. Compare the confusion matrices, or use a statistic that is less sensitive to prevalence.
Treating agreement as correctness
Two annotators trained on the same guideline share its misreadings, so they agree and are both wrong. A high kappa says the annotators are consistent with each other and nothing about whether the labels follow the guideline. Correctness needs an audit against the guideline on a sample drawn from the batch.
Computing kappa on a sample that was not designed for it
The expected agreement is estimated from the marginals of the sample, so a sample deliberately enriched with rare classes will produce a different kappa than the full batch would. If the sample was drawn to contain examples of each class, the kappa describes that sample. Report how the items were selected with the score.
Ignoring the confidence interval
Kappa measured on a hundred items carries an interval wide enough that 0.44 and 0.62 may not be distinguishable. Reporting a bare kappa invites comparisons that the sample size does not support. Compute the interval, or at least state the sample size next to the number so the reader can judge.
When to buy the data instead of building the pipeline
Compute kappa yourself; it is a few lines and the number has to be read next to your own label distribution. Buy the annotation when the class list needs domain knowledge or the volume exceeds your team, and write the agreement target into the agreement with the label distribution it applies to. A supplier who quotes a kappa without the distribution is quoting a number that cannot be interpreted. We report agreement with the confusion matrix and the label distribution, per class where the classes are skewed.
Need data for Cohen's Kappa?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.