Annotation Quality
Short answer. Annotation quality is measured on a sample drawn from the delivered batch and compared against the guideline, not against another annotator. The number that matters is the error rate with an interval around it, because a sample of fifty cannot distinguish a three percent error rate from a thirteen percent one. Auditing fifty items out of ten thousand produces an interval twelve points wide, which is not evidence of anything. Deciding the sample size from the error rate you need to rule out is the step that makes an audit worth running.
How to do it
An audit is not an agreement measurement
Agreement asks whether two annotators produce the same labels, and it is a property of the guideline and the annotators. An audit asks whether the delivered labels follow the guideline, and it is a property of the batch. A delivery can have high agreement between two annotators who both misread the same instruction, so agreement alone does not tell you the labels are right.
The audit needs a reference to compare against, and the cheapest honest reference is a second annotation by someone senior enough to adjudicate, on a sample drawn by the person running the project. The sample has to be random and drawn before the audit starts, otherwise the sample is chosen by whoever wants the number to look good.
Why the sample size decides whether the audit means anything
An error rate measured on a sample is an estimate, and the width of the interval around it depends mostly on the sample size. Fifty items gives an interval around twelve points wide at a three percent rate, which means the audit cannot tell a good batch from a bad one. Four hundred items narrows it to under four points, which is enough to act on.
The right way to choose the size is to start from the question. If the requirement is that the error rate is below five percent, the sample has to be large enough that the upper end of the interval falls below five when the observed rate is three. That is roughly a thousand items, not a hundred, and it is worth knowing before promising a client that the audit will prove the requirement.
What to record with the number
An error rate without its sample size and its interval is not interpretable. Record the size of the batch, the size of the sample, how the sample was drawn, who produced the reference, and the guideline version the audit was run against. Without the last one, a later audit against a revised guideline cannot be compared with the first.
Break the number down by class and by annotator. A batch with an overall error rate of two percent and one class at eleven percent passes an overall gate and fails the product in the class that matters. The breakdown is also what tells you whether the problem is one person, one class, or the guideline.
Code
Compute the interval around an audit result for several sample sizes, and work out what sample size is needed to prove an error rate is below a threshold.
import math
def wilson(k, n, z=1.96):
if n == 0:
return 0.0, 1.0
p = k / n
d = 1 + z * z / n
c = (p + z * z / (2 * n)) / d
h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
return max(0.0, c - h), min(1.0, c + h)
print("auditing a batch of 10,000 items, true error rate 3 percent")
print()
print(" sample errors measured 95 percent interval width")
for n in (50, 100, 385, 1000, 2000):
k = round(0.03 * n)
lo, hi = wilson(k, n)
print(" %6d %6d %7.1f%% %5.1f%% to %5.1f%% %5.1f pt"
% (n, k, 100.0 * k / n, 100 * lo, 100 * hi, 100 * (hi - lo)))
print()
print("the batch is 10,000 items. auditing 50 of them and finding 4 percent leaves")
print("an interval 12 points wide, so the same audit cannot rule out 13.5 percent.")
print()
print("what the same sample sizes can detect, if you need to prove the rate is under 5 percent")
print()
for n in (50, 100, 385, 1000):
k = round(0.03 * n)
lo, hi = wilson(k, n)
verdict = "cannot prove it" if hi > 0.05 else "proves it"
print(" sample %5d -> upper bound %5.1f%% %s" % (n, 100 * hi, verdict))
print()
print("a per-item audit is not the same as an agreement measurement. agreement asks")
print("whether two annotators match; an audit asks whether the delivered labels follow")
print("the guideline, and it needs a sample drawn from the delivered batch.")- Standard library only. The interval is the Wilson score interval, which behaves better than the textbook normal approximation when the rate is small or the sample is small.
- Expected: a sample of 50 with 2 errors reports 4.0 percent with an interval from 1.1 to 13.5 percent. A sample of 1000 with 30 errors reports 3.0 percent with an interval from 2.1 to 4.3 percent.
- The second table is the one to use when writing an acceptance criterion. A sample of 385 is the familiar figure for a five percent margin at a fifty percent rate, and it is not enough here: at a three percent observed rate the upper bound is 5.4 percent, so it cannot prove the rate is under five. A sample of 1000 gives 4.3 percent and does.
- Set the true rate to 0.10 in the first line and rerun to see the intervals widen, because the width of an interval around a proportion is largest near fifty percent.
Where this goes wrong
Auditing a sample the annotators chose
If the people who produced the batch pick the items to be audited, they pick the easy ones and the resulting number describes the sample rather than the batch. The sample has to be drawn at random by the person running the project, before the audit starts, and the draw has to be recorded. This is the most common way an audit becomes decoration.
Reporting a rate without the interval
A rate of three percent from a sample of fifty and a rate of three percent from a sample of two thousand are different claims, and reporting only the point estimate invites the reader to treat them as the same. Report the sample size and the interval alongside the rate, and if the interval crosses the acceptance threshold, say that the audit is inconclusive rather than that it passed.
Averaging over classes
An overall error rate hides the class that is broken. A batch at two percent overall with one class at eleven percent passes an overall gate and fails wherever that class is used. Compute the rate per class and write the classes the product depends on into the acceptance criteria on their own.
Auditing against a different guideline version
If the guideline was revised after the batch was produced, an audit against the current version measures the guideline change rather than the batch. Record the version the batch was produced against and audit against that one, then separately assess what the revision implies for the data already delivered.
When to buy the data instead of building the pipeline
Run the audit yourself, because the sample has to be yours and the reference has to be someone whose judgement you trust. Buy the annotation when the volume is beyond your team or the domain needs knowledge you do not have, and write the audit into the agreement: sample size, how it is drawn, who produces the reference, and the threshold. A supplier who will not accept an audit on a random sample is telling you something. We accept a random-sample audit against the guideline version the batch was produced with, and report the error rate per class with its interval.
Need data for Annotation Quality?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.