Data Labeling vs Annotation
Short answer. Labeling attaches one value to a whole item: this review is negative, this image is a cat. Annotation attaches a variable number of structured objects to an item: these two spans are organizations, these three boxes are people. The distinction is not terminology, it is the cost model. Labeling has one unit per item, so the price is per item. Annotation has a density that varies per item, so a price quoted per item loses money on the dense ones and overcharges on the empty ones.
How to do it
One value per item against a variable structure
A labeling task has a fixed shape: every item gets exactly one value from a closed set. That makes the row count equal to the unit count, which makes progress tracking, pricing and quality sampling all straightforward. It also means the output can be stored as one column.
An annotation task has a shape that varies per item. A short review may contain no entity at all, and a long one may contain a dozen. The unit count is the number of objects, not the number of items, and it is not known until the work is done. Every estimate made before that point is a guess about density.
- labeling: one value per item, closed set, unit count equals item count
- annotation: zero or more objects per item, unit count unknown until annotated
- the empty items are real work and still have to be read
Why the price per item breaks
A quote written per item assumes a density. If the actual density is higher, the annotator spends more time per item than the price covers, and the work either slows down or the quality falls, usually both. If the density is lower, the buyer pays for time that was not spent.
The fix is to price annotation per object with a separate rate for reading, or to measure the density on a pilot and quote against that measurement. Either way the density is the number that has to be agreed before the batch starts, and it is measured on a random sample of the actual data rather than estimated from a description.
What changes on the quality side
For labeling, quality is a per-item question: is this the right value. For annotation, quality has two parts that fail independently. The spans that were drawn can be wrong, and the spans that were not drawn can be missing. Recall failures are invisible in a sample that only checks the spans that exist, which is why an annotation audit has to start from the source item rather than from the delivered objects.
This is also why agreement is measured differently. Two annotators who both find three entities agree on precision and can still both miss a fourth, and their agreement score will not show it. The audit against the source text is what catches the shared blind spot.
Code
Compare the two task shapes on the same six items, show how the density distribution breaks a per-item price, and compute what the same batch costs under each model.
from collections import Counter
# the same six items; the two task shapes have completely different cost models
labeling = [
{"id": 1, "text": "box was crushed", "label": "negative"},
{"id": 2, "text": "arrived early", "label": "positive"},
{"id": 3, "text": "fine, nothing special", "label": "neutral"},
{"id": 4, "text": "broken on arrival", "label": "negative"},
{"id": 5, "text": "great, would buy again", "label": "positive"},
{"id": 6, "text": "it is a box", "label": "neutral"},
]
annotation = [
{"id": 1, "text": "box was crushed", "spans": [("box", "OBJ"), ("crushed", "STATE")]},
{"id": 2, "text": "arrived early", "spans": [("arrived", "ACTION"), ("early", "TIME")]},
{"id": 3, "text": "fine, nothing special", "spans": []},
{"id": 4, "text": "broken on arrival", "spans": [("broken", "STATE"), ("arrival", "TIME")]},
{"id": 5, "text": "great, would buy again", "spans": [("great", "SENT"), ("buy", "ACTION"), ("again", "TIME")]},
{"id": 6, "text": "it is a box", "spans": []},
]
print("labeling: %d items, %d labels, %.2f labels per item"
% (len(labeling), len(labeling), 1.0))
print(" every item has exactly one label, so the row count is the unit count")
print()
total = sum(len(r["spans"]) for r in annotation)
empty = sum(1 for r in annotation if not r["spans"])
print("annotation: %d items, %d spans, %.2f spans per item"
% (len(annotation), total, total / len(annotation)))
print(" %d of %d items carry no span at all (%.0f%%)"
% (empty, len(annotation), 100.0 * empty / len(annotation)))
print(" the longest item carries %d spans" % max(len(r["spans"]) for r in annotation))
print()
print("cost model, at 0.04 per item and 0.03 per span:")
per_item = 0.04 * len(labeling)
per_span = 0.04 * len(annotation) + 0.03 * total
print(" priced per item %.2f" % per_item)
print(" priced per span %.2f (%.1f%% more)" % (per_span, 100.0 * (per_span / per_item - 1)))
print()
print("a quote for the second task written per item prices it at the first rate and")
print("loses money on the dense items. ask for the span density, not just the item count.")
dist = Counter(len(r["spans"]) for r in annotation)
print()
print("span count distribution: %s" % dict(sorted(dist.items())))
print("mean %.2f, and %.0f%% of items are at or below the mean"
% (total / len(annotation),
100.0 * sum(v for k, v in dist.items() if k <= total / len(annotation)) / len(annotation)))- Standard library only.
- Expected: the labeling set is 6 items and 6 units, so the unit count equals the item count. The annotation set is 6 items and 9 spans, with 2 of the 6 items carrying nothing and one carrying 3.
- At the rates used here the per-item price is 0.24 and the per-span price is 0.51, which is 112.5 percent more. The exact figures are made up, but the direction is not: a dense annotation task costs several times a labeling task on the same item count.
- The last two lines are the part that bites. The mean is 1.50 spans per item and 33 percent of items are at or below it, so a price derived from the mean overpays for a third of the batch and underpays for the rest. That is why the density distribution, not the mean, is what belongs in the quote.
Where this goes wrong
Quoting an annotation task per item
The price assumes a density that has not been measured. If the real density is higher, the annotator either slows down or cuts corners, and both show up as a quality problem rather than as the pricing problem they are. Measure the density on a random pilot sample and quote per object with a separate reading rate.
Treating empty items as free
An item with no entities still has to be read to know it has none. In a batch where a third of the items are empty, that is a third of the reading time with no objects produced, and a per-object price alone does not cover it. Price the reading separately, or quote per item with a per-object supplement.
Auditing only the objects that were produced
Checking the spans that exist measures precision and says nothing about the ones that were missed. Two annotators can agree closely and share the same blind spot, so agreement does not catch it either. An annotation audit has to start from the source item and ask what should have been marked, not from the delivered list.
Choosing the task shape by the output you want
Labeling is cheaper and easier to control, so there is pressure to frame a structured problem as a labeling one. If the product needs to count or locate objects, a single label per item cannot express the answer and the model will not learn it. Decide the task shape from what the model has to output, then price it.
When to buy the data instead of building the pipeline
Decide the task shape yourself, because it follows from what the model must output. Buy the work when the volume is beyond your team or the domain needs knowledge you do not have, and specify it in the terms that make it priceable: the item count, the measured object density, the guideline, and the audit. A quote that names only an item count is a quote with an unmeasured assumption inside it. We run a pilot to measure the density before quoting, and price the reading and the objects separately.
Need data for Data Labeling vs Annotation?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.