Instance Segmentation
Short answer. Instance segmentation gives every pixel of every object a label that is unique to that object, so two people standing together get different labels rather than one shared person label. The output is a label map plus an instance table, and the two have to agree: a mask with no row in the table is invisible to the evaluator, and a row with no pixels is an error the loader will not report. Touching objects, occlusion, and two objects of the same class side by side are what the guideline has to settle, because the annotation decides whether the model can ever learn to separate them.
How to do it
Instances versus classes
Semantic segmentation answers which class each pixel belongs to and merges all objects of a class into one region. Instance segmentation answers which object each pixel belongs to, and the class is a property of the instance. A semantic mask for a street scene has one region labelled person no matter how many people are in it, so it cannot support counting and cannot support tracking.
The practical difference for annotation is that instances have to be separated at the point where they touch. That is a judgement call: two cars parked bumper to bumper, two people in a crowd, a stack of boxes. The guideline has to give a rule, because an annotator who merges them produces one instance where the product needs two, and the error is invisible in any summary statistic.
Overlapping objects and drawing order
Masks are stored one per object and they overlap, because a person behind a table owns the pixels of their visible torso and the table owns the pixels in front of it. That is different from a label map, where each pixel has exactly one value, and the conversion between the two is a decision: whichever mask is painted last wins, so the order has to be fixed by the rule that the nearer object is painted later.
The evaluator works on the per-object masks, not on the flattened map, so a dataset stored as a single label map has already lost the overlapping regions. Recovering them is not possible. If the source annotation was per-object masks, keep them per-object through the whole pipeline and flatten only for display.
How the metric treats a mask
Each predicted mask is matched to a ground truth mask of the same class, and the match is decided by the IoU between the masks rather than between their bounding boxes. A mask that covers the right box but the wrong pixels fails, which is the point of the task. The score is then averaged over classes the same way detection is, so a rare class counts as much as a common one.
Because the match is per instance, the number of instances in the ground truth matters and has to be right. A merged instance reduces the ground truth count, and a split instance raises it, and both change the score in ways that look like model behaviour. Check the instance count per class against the source before accepting a delivery.
Code
Build a small instance label map, extract per-instance masks and areas, and measure the IoU between two annotators on the same image.
import numpy as np
# 0 = background. Two annotators label the same 6x6 crop.
a = np.array([
[0, 0, 0, 0, 0, 0],
[0, 1, 1, 0, 2, 2],
[0, 1, 1, 0, 2, 2],
[0, 1, 1, 1, 0, 0],
[0, 0, 0, 0, 0, 0],
])
b = np.array([
[0, 0, 0, 0, 0, 0],
[0, 1, 1, 0, 2, 2],
[0, 1, 1, 0, 2, 2],
[0, 1, 1, 1, 2, 0],
[0, 0, 0, 0, 0, 0],
])
for name, m in (("annotator A", a), ("annotator B", b)):
ids = [int(i) for i in np.unique(m) if i != 0]
print(name, "instances:", ids)
for i in ids:
mask = m == i
ys, xs = np.nonzero(mask)
print(" instance %d: %4d px, bbox x %d-%d y %d-%d"
% (i, int(mask.sum()), xs.min(), xs.max(), ys.min(), ys.max()))
def mask_iou(m1, m2):
inter = int(np.logical_and(m1, m2).sum())
union = int(np.logical_or(m1, m2).sum())
return inter / float(union) if union else 0.0
print()
print("instance 1 IoU between annotators: %.3f" % mask_iou(a == 1, b == 1))
print("instance 2 IoU between annotators: %.3f" % mask_iou(a == 2, b == 2))
print()
print("one extra pixel on instance 2 costs 0.200 on that instance and 0.000 on instance 1")
print("instance count agrees, per-instance agreement does not")- pip install numpy
- Expected: both annotators report two instances with the same bounding boxes. Instance 1 scores 1.000 and instance 2 scores 0.800. The bounding boxes are identical, so any metric computed on boxes would call these two annotations the same.
- That is the reason the mask is evaluated and not the box. A single disputed pixel is invisible in the box metric and visible in the mask metric, and on a real object the disputed pixels are along the whole outline.
Where this goes wrong
Merging touching objects of the same class
Two people standing shoulder to shoulder are easy to label as one region and hard to separate, and the merged label looks clean. The model trained on it can never learn the boundary, so counting and tracking both fail in exactly the crowded scenes where they matter. Give the annotators a rule for touching objects and check the instance count per class on the pilot sample.
Delivering a flattened label map
A label map stores one class per pixel and cannot represent two overlapping objects. If the delivery is a label map and the requirement was instance masks, the overlapping regions are gone and no post-processing recovers them. State the storage form in the specification: per-object masks, a run-length encoding of them, or polygons, and say whether overlaps are preserved.
Ignoring holes and thin structures
A mask with a hole, such as the gap between a bicycle frame and its rider, is a real part of the object and is easy to fill in by accident when the mask is stored as a polygon. Polygon formats need either multiple contours per object or an explicit rule that holes are not represented, and the evaluator has to know which. Check a few objects with genuine holes in the pilot.
Judging quality by looking at the masks
Masks look convincing at a glance because the outline follows the object. The failures are in the instance count and in the boundaries between touching objects, and neither is visible when you look at one image at a time. Measure per-instance IoU between two annotators on a sample, and check the instance count per class against the source.
When to buy the data instead of building the pipeline
Build the mask viewer and the evaluator yourself, since both encode your definition of an object. Buy the annotation when the class list is long, the objects touch, or the volume is beyond what your own team can sustain at a consistent standard. Ask for per-object masks rather than a label map, the rule for touching objects, and a pilot with per-class instance counts and agreement. We produce instance masks with the overlap rule and the instance counts documented per class, checked against the source before delivery.
Need data for Instance Segmentation?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.