Panoptic Segmentation
Short answer. Panoptic segmentation gives every pixel exactly one label, and that label is either a class for background surfaces or an instance for countable objects. It merges two tasks that were previously separate: semantic segmentation, which knows a pixel is road but not which road, and instance segmentation, which knows a car is a separate car but has nothing to say about the road. The merge rule is what the task is: things win over stuff, and the output is a single label map with no overlap anywhere.
How to do it
Two label sets forced into one map
Semantic segmentation labels every pixel with a class, so two cars parked next to each other become one region of the class car. Instance segmentation labels each countable object separately but typically leaves the background unlabelled. Panoptic segmentation requires both at once, on one map, with no pixel claimed twice.
The classes split into stuff, which is uncountable and has no instances, and things, which are countable and each get their own id. Road, sky, vegetation and wall are stuff. Car, person, sign and cyclist are things. The output is a single array of ids where a negative value means void and every other value resolves to either a stuff class or a thing instance.
- stuff: uncountable surfaces, one region per class, no instances
- things: countable objects, one region per object, each with its own id
- every pixel gets exactly one id, so nothing overlaps and nothing is left out
The merge rule is the hard part
If the two source models disagree about a pixel, the merged map has to choose. The common rule is that things beat stuff, so a pixel that semantic segmentation calls road and instance segmentation assigns to a car becomes part of the car. That is right when the instance model is more reliable at object boundaries and wrong when it hallucinates.
A second problem is that instance models trained alone often produce overlapping masks. Two predicted cars can claim the same pixels, and the merged map cannot represent that. The usual fix is to order instances by score or by area and let the earlier one keep the contested pixels, which is a decision made outside the model and has to be recorded. Two pipelines that both call themselves panoptic can differ by this rule alone.
Panoptic quality, and why the threshold travels with it
Panoptic quality is computed per class and averaged. For each class, predicted regions are matched to ground truth regions by intersection over union above a threshold, usually 0.5, and each region can be matched once. The metric is the sum of the matched intersections over the number of matched pairs plus half the unmatched predictions plus half the unmatched ground truths.
That expression factors into a recognition term and a segmentation term: how many objects were found, and how well the matched ones overlap. A pipeline can score well on one and badly on the other, and the two numbers are what tell you which to fix. The threshold has to be reported with the score, because an object matched at 0.6 counts at 0.5 and does not at 0.75, and the same predictions then produce two different numbers.
Code
Compute panoptic quality on a small label map, split it into its recognition and segmentation parts, and show how much the score moves when the match threshold changes.
# 8x8 grid, panoptic: every pixel belongs to exactly one thing
# 0 = background (stuff), positive = instance id
GT = [
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 1, 1, 1, 1, 0, 0, 0],
[0, 1, 1, 1, 1, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 2, 0],
[0, 0, 0, 0, 0, 0, 2, 0],
[0, 0, 0, 0, 0, 0, 2, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
]
# prediction: car matches exactly, person is one pixel short, plus an extra instance 3
# that does not exist in the ground truth
PRED = [
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 1, 1, 1, 1, 0, 3, 0],
[0, 1, 1, 1, 1, 0, 3, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 2, 0],
[0, 0, 0, 0, 0, 0, 2, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
]
def pixels(m):
d = {}
for y, row in enumerate(m):
for x, v in enumerate(row):
if v:
d.setdefault(v, set()).add((y, x))
return d
def iou(a, b):
inter = len(a & b)
return inter / len(a | b) if (a | b) else 0.0
g, p = pixels(GT), pixels(PRED)
print("ground truth instances: %s" % {k: len(v) for k, v in sorted(g.items())})
print("predicted instances: %s" % {k: len(v) for k, v in sorted(p.items())})
print()
print("each pixel belongs to exactly one instance. there is no overlap to resolve,")
print("which is the whole point of panoptic: things and stuff share one label map.")
print()
pairs = sorted(((iou(g[i], p[j]), i, j) for i in g for j in p), reverse=True)
matched_g, matched_p, tps = set(), set(), []
for v, i, j in pairs:
if v > 0.5 and i not in matched_g and j not in matched_p:
matched_g.add(i)
matched_p.add(j)
tps.append((i, j, v))
print("IoU above 0.5:")
for i, j, v in tps:
print(" gt %d <-> pred %d IoU %.3f" % (i, j, v))
fp = len(p) - len(matched_p)
fn = len(g) - len(matched_g)
print(" false positives %d, false negatives %d" % (fp, fn))
sq = sum(v for _, _, v in tps) / len(tps) if tps else 0.0
rq = len(tps) / (len(tps) + 0.5 * fp + 0.5 * fn)
pq = sum(v for _, _, v in tps) / (len(tps) + 0.5 * fp + 0.5 * fn)
print()
print("PQ %.3f = SQ %.3f (how well matched pairs overlap) x RQ %.3f (how many were found)"
% (pq, sq, rq))
print()
for thr in (0.5, 0.75):
mg, mp, t = set(), set(), []
for v, i, j in pairs:
if v > thr and i not in mg and j not in mp:
mg.add(i)
mp.add(j)
t.append(v)
f_p, f_n = len(p) - len(mp), len(g) - len(mg)
q = sum(t) / (len(t) + 0.5 * f_p + 0.5 * f_n)
print(" threshold %.2f -> PQ %.3f (TP %d, FP %d, FN %d)" % (thr, q, len(t), f_p, f_n))
print()
print("raising the threshold from 0.50 to 0.75 turns the person from a match into")
print("a miss plus a false positive, and PQ drops from 0.667 to 0.400. report the")
print("threshold with the number; a bare PQ is not comparable across papers.")- Standard library only.
- Expected: the ground truth has two instances with 8 and 3 pixels, the prediction has three with 8, 2 and 2. The car matches exactly at 1.000, the person matches at 0.667 because one pixel is missing, and the third prediction has no counterpart.
- Expected: PQ is 0.667, which factors into SQ 0.833 and RQ 0.800. The recognition term is below one because of the extra prediction, and the segmentation term is below one because of the missing pixel. A pipeline that scores well on one and badly on the other needs a different fix, which is why the split is worth printing.
- Expected: at a threshold of 0.75 the person at 0.667 stops matching, and the score drops from 0.667 to 0.400. Set the threshold to 0.9 and it drops again, because even the car is then compared against a stricter bar.
- The pixels function treats every nonzero value as an instance. In a real pipeline the stuff classes need their own ids and the merge rule that decides contested pixels has to be applied before this point.
Where this goes wrong
Reporting a panoptic score without the match threshold
The threshold is part of the definition, not a tuning knob. An object that overlaps its ground truth by 0.6 counts at 0.5 and does not at 0.75, and the same predictions then produce two different scores. A number quoted without its threshold cannot be compared with anything, including an earlier run of the same pipeline.
Leaving the overlap rule undocumented
Instance models produce overlapping masks, and a single label map cannot hold them. Whether the larger instance wins, or the higher scoring one, or the one that appears first, changes the output on every contested pixel. Two teams can both report panoptic results and differ only by this rule, so it belongs in the documentation next to the model version.
Judging the pipeline from one number
Panoptic quality hides whether the failure is finding objects or outlining them. A pipeline that finds every object with sloppy boundaries and one that outlines a few objects precisely can land on the same score. Print the recognition and segmentation terms separately, and look at them per class, because the small classes are where the score usually collapses.
Annotating stuff and things in one pass without a rule
A single annotator drawing cars and road at the same time will leave pixels unassigned at the boundary or assign them twice. The guideline needs an explicit order and an explicit rule for the contested strip, or the ground truth itself will be inconsistent in a way that caps how well any model can score.
When to buy the data instead of building the pipeline
Decide the class list and the overlap rule yourself, because they define the task and no supplier can guess them. Buy the annotation when the class list is long, when the boundary rule needs domain knowledge, or when the volume exceeds what your team can draw. Ask for the annotation guideline, the overlap rule, and a per-class instance count, because a panoptic dataset with two objects in a class cannot train or evaluate that class. We deliver per-pixel label maps with the class list, the overlap rule and the per-class counts documented.
Need data for Panoptic Segmentation?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.