Intersection over Union (IoU)
Short answer. IoU is the area where two boxes overlap divided by the area they cover together: IoU = overlap / (area A + area B - overlap). It runs from 0 for boxes that do not touch to 1 for identical boxes, and it is the unit every detection threshold is written in. A prediction counts as correct at IoU 0.5 in most benchmarks, which means a box can be wrong by about a fifth of its own size and still score. IoU is scale invariant: a small box and a large box with the same relative error score the same, and that is exactly why a single threshold is a bad fit for objects of very different sizes.
How to do it
Compute it from two corner pairs
Convert both boxes to corner form first, (x1, y1) top-left and (x2, y2) bottom-right, with the origin at the top-left of the image. Every format stores something different: COCO stores top-left plus width and height, YOLO stores the centre plus width and height normalized by the image size, Pascal VOC stores the corner pair. The IoU formula below assumes corners, so the conversion is where most wrong numbers come from.
The intersection is computed per axis and clamped at zero, because boxes that do not overlap produce a negative width or height and a negative area would poison the result. The union is the sum of the two areas minus the intersection, which counts the shared region once.
- inter_width = max(0, min(a.x2, b.x2) - max(a.x1, b.x1))
- inter_height = max(0, min(a.y2, b.y2) - max(a.y1, b.y1))
- inter = inter_width * inter_height
- union = area_a + area_b - inter
- IoU = inter / union
The threshold is a data decision, not a default
The same model reports very different accuracy at IoU 0.5 and at IoU 0.75, and neither number is wrong. On the COCO benchmark the gap between the two is routinely ten points or more, because a looser threshold forgives a box that is roughly a fifth too large in each direction. If your product draws a box on a screen for a human to check, 0.5 is usually generous enough. If the box is used to crop a region that is then classified, the crop includes background when the box is loose, and the classification accuracy falls in a way that never shows up in the detection metric.
For annotation quality control the threshold has to be set from measured human agreement, not from the benchmark convention. Two trained annotators drawing the same object usually land between 0.7 and 0.8 IoU, so a rule that rejects anything under 0.9 rejects most honest work, and a rule that accepts anything over 0.3 accepts boxes that cover the wrong object. Measure your own annotators on a shared sample before you write the acceptance number into a contract.
What IoU does not measure
IoU has no gradient when two boxes do not overlap, so it cannot be used as a training loss directly; that is why the loss functions in use are variants that add a penalty for distance and for aspect ratio. It is worth knowing because a spec that says the model should be optimized for IoU is describing a metric, not a loss.
IoU also ignores class and ignores count. A prediction that finds the right box with the wrong label has an IoU of 1.0 and is still a miss, and a model that emits five overlapping boxes for one object can have a perfect IoU on each of them. That is why the detection metric is mean average precision over matched pairs, which pairs IoU with a class check and a ranking, rather than an average IoU.
Code
Compute IoU for a pair, build the full prediction-by-ground-truth IoU matrix, and show that the same relative error costs the same IoU at any object size.
import numpy as np
def iou(a, b):
"""a, b = (x1, y1, x2, y2) in pixels, origin at the top-left corner."""
ix1, iy1 = max(a[0], b[0]), max(a[1], b[1])
ix2, iy2 = min(a[2], b[2]), min(a[3], b[3])
inter = max(0.0, ix2 - ix1) * max(0.0, iy2 - iy1)
area_a = (a[2] - a[0]) * (a[3] - a[1])
area_b = (b[2] - b[0]) * (b[3] - b[1])
union = area_a + area_b - inter
return inter / union if union > 0 else 0.0
gt = [(10, 10, 110, 110), (200, 40, 240, 80)]
pred = [(14, 16, 108, 112), (196, 36, 244, 84), (500, 500, 540, 540)]
M = np.zeros((len(pred), len(gt)))
for i, p in enumerate(pred):
for j, g in enumerate(gt):
M[i, j] = iou(p, g)
print("IoU matrix, rows = predictions, cols = ground truth")
print(np.round(M, 3))
print()
for size in (20, 200):
g = (0, 0, size, size)
p = (size * 0.1, 0, size * 1.1, size)
print("box %4d px, shifted %.0f px on each side -> IoU %.3f"
% (size, size * 0.1, iou(p, g)))- pip install numpy
- Expected: the matrix is 3 by 2. Row 0 matches ground truth 0 at about 0.87 and ground truth 1 at 0.0. Row 2, the box at the far corner, is 0.0 across the row, which is what a false positive looks like before any threshold is applied.
- The last two lines print the same IoU, 0.818, for a 20 pixel box and a 200 pixel box. Both were widened by ten percent on each side. IoU cannot tell them apart, but the small box moved by 2 pixels and the large one by 20, and only one of those is visible to a human reviewer.
- The matrix is the input to matching. A real evaluator sorts predictions by confidence, greedily assigns each to the best ground truth above the threshold, and marks the rest as false positives.
Where this goes wrong
Mixing up the box convention
Feeding a COCO box straight into a corner-based IoU function reads width and height as x2 and y2. On a 1920 by 1080 image a box at (412, 233, 96, 210) becomes a corner box from (412, 233) to (96, 210), which has negative area. Depending on the clamp the result is either 0.0 everywhere or a silently mirrored box. Convert once, at the boundary, and name the variables so the convention is visible in the code.
One threshold for every object size
A 20 pixel object at IoU 0.5 allows roughly a 3 pixel error. A 200 pixel object at the same threshold allows 30 pixels. A spec that says boxes must be accurate to IoU 0.5 is therefore far stricter on small objects than on large ones, and a dataset of mostly large objects will pass it while the small ones are unusable. If the product depends on small objects, write the tolerance in pixels and let the IoU threshold follow.
Judging annotation quality at the benchmark threshold
IoU 0.5 comes from detection evaluation, where the question is whether a model found the object. It is not a quality bar for the annotation itself. Two people tracing the same car agree at around 0.75 to 0.85, and the number drops when the object is partly hidden or when the rule about whether to include the visible part or the estimated full extent is not written down. Set the acceptance threshold from your own measured agreement, and record the rule alongside it.
Averaging IoU over predictions instead of matching them
The mean IoU over all predictions rewards a model that emits many boxes, because each one is scored against its best match and the extras add rows to the average. Detection metrics match first and count unmatched predictions as errors, which is why a model with a lower mean IoU can have a higher average precision. If you report a mean IoU, say which ground truth each prediction was matched to and how many went unmatched.
When to buy the data instead of building the pipeline
Build the IoU function yourself; it is six lines and you should own the convention. Buy the data when the number depends on annotation quality rather than on your code: a corpus where the box rule is written down, applied by annotators who were measured against each other, and reported with an agreement figure you can compare against your own. That is the part that cannot be derived from an open dataset, because open datasets publish the boxes but not the rule that produced them. We assemble detection corpora with the guideline, the measured inter-annotator agreement, and the class list documented, so the IoU your model reaches in production is the one the specification predicted.
Need data for Intersection over Union (IoU)?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.