Mean Average Precision (mAP)
Short answer. mAP is the mean of average precision across classes. Average precision is the area under the precision-recall curve for one class, computed by walking its predictions from the most confident to the least and tracking precision at each step in recall. It is the headline detection number because it scores the ranking, not one operating point. The value is only comparable to another computed the same way: the IoU threshold, the interpolation rule, and whether classes are averaged equally or by instance count all change it.
How to do it
From ranked predictions to a precision-recall curve
Sort every prediction for one class by confidence, highest first. Walk the list and keep two running totals: how many ground truth objects have been matched so far, and how many predictions have been seen. Precision at each step is matched divided by seen. Recall is matched divided by the number of ground truth objects in that class.
A prediction counts as matched when its IoU with an unmatched ground truth of the same class is above the threshold. Each ground truth can be matched once, so a second prediction on the same object is a false positive, which is how the metric penalizes duplicate boxes.
Why the curve is made monotone before the area is taken
Raw precision jitters: a lucky match can raise it after a miss lowered it, so the sequence is not monotone and the area under it is not a meaningful summary. The standard fix is to replace each precision value with the maximum precision at any higher recall, which produces a non-increasing curve. The area is then taken under that envelope.
The older convention sampled the envelope at eleven fixed recall points from 0.0 to 1.0 and averaged them. The current one integrates over all recall steps and is the reason numbers from papers before and after the change are not comparable. When someone quotes a mAP, the interpolation rule matters as much as the IoU threshold.
The conventions that decide the number
Three settings change the result and are rarely stated: the IoU threshold used for matching, whether the average over classes is a plain mean or weighted by the number of instances, and whether the maximum number of predictions per image is capped. A plain mean over classes rewards rare classes as much as common ones, which is usually what you want if the rare class is the product. A weighted average is dominated by whichever class has the most boxes.
For a dataset you are buying, the useful version of the question is which of these settings the acceptance criteria assume. A contract that says the model must reach mAP 0.45 is underspecified until the threshold and the averaging rule are written down, and a vendor who does not know them cannot be held to the number either.
Code
Compute average precision from a ranked prediction list with all-point interpolation, for two classes, and average them into mAP.
import numpy as np
# Per class: predictions ranked by confidence, high to low. 1 = hit, 0 = false positive
DETS = {
"person": [1, 1, 0, 1, 0, 1, 1],
"car": [1, 0, 1, 1, 1],
}
GT_COUNT = {"person": 5, "car": 4}
def precision_recall(hits, n_gt):
hits = np.asarray(hits, dtype=float)
tp = np.cumsum(hits)
fp = np.cumsum(1.0 - hits)
recall = tp / n_gt
precision = tp / np.maximum(tp + fp, 1e-9)
return precision, recall
def ap_all_point(hits, n_gt):
p, r = precision_recall(hits, n_gt)
p = np.maximum.accumulate(p[::-1])[::-1] # make the curve monotonically non-increasing
mrec = np.concatenate(([0.0], r, [r[-1]]))
mpre = np.concatenate(([0.0], p, [0.0]))
idx = np.where(mrec[1:] != mrec[:-1])[0]
return float(np.sum((mrec[idx + 1] - mrec[idx]) * mpre[idx + 1]))
aps = {}
for cls, hits in DETS.items():
p, r = precision_recall(hits, GT_COUNT[cls])
aps[cls] = ap_all_point(hits, GT_COUNT[cls])
print(cls)
print(" precision", np.round(p, 3))
print(" recall ", np.round(r, 3))
print(" AP = %.3f" % aps[cls])
print()
print("mAP = %.3f" % (sum(aps.values()) / len(aps)))- pip install numpy
- Expected: person scores 0.836 and car scores 0.850, so mAP prints 0.843. The person list has two misses and still scores well, because the envelope only looks at the best precision available at each recall, and the misses come late enough that the envelope stays high across most of the recall range.
- Move one of the zeros in the person list to the front and rerun. The score falls, even though the number of hits and misses is unchanged. That is the ranking effect mAP exists to measure, and it is why a detection metric cannot be replaced by counting correct boxes.
- If a class has more hits than ground truth objects, recall goes above 1.0 and the number is meaningless. That means the ground truth count is wrong, not that the model is good.
Where this goes wrong
Comparing mAP numbers computed at different IoU thresholds
A mAP at IoU 0.5 and a mAP at IoU 0.75 differ by ten points or more on the same model, and the benchmark that reports a single number usually averages several thresholds together. Two vendors quoting 0.52 may be reporting three different quantities. Ask for the threshold list and the interpolation rule before you compare, and write both into the acceptance criteria.
Averaging classes equally when the classes are not equal
A plain mean gives a class with twenty boxes in the whole test set the same weight as a class with twenty thousand. That is often correct, because it keeps the model from ignoring the rare class. It also means a single tiny class with a lucky score can move the headline number. Report per-class average precision next to the mean, and look at the spread before you accept the total.
Reporting mAP for a model that was tuned on the test set
Detection numbers are easy to inflate by adjusting the confidence threshold or the non-maximum-suppression setting against the evaluation set. The result is a number that describes the tuning, not the model. If you are buying data to reach a target, ask that the test set be fixed and held out before tuning starts, and that the same threshold be used for every model compared.
Treating mAP as a measure of annotation quality
A low mAP can come from a hard task, a weak model, or a test set whose boxes follow a different convention from the training set. Before blaming the model, check the annotation: if the training boxes cover the visible extent and the test boxes cover the estimated full extent, the ceiling is set by that disagreement and no amount of training will pass it. The agreement figure between annotators is the number that tells you which of the two you are looking at.
When to buy the data instead of building the pipeline
Compute the metric yourself; the evaluator is part of your product and it has to match your acceptance criteria. Buy the data when you need per-class coverage you cannot collect: rare classes, small objects, and the conditions your product will meet but your own camera has never seen. Ask for the class list with instance counts and the annotation rule for partly hidden objects, because those two documents decide whether the mAP you measure transfers. We deliver detection corpora with the class list, the per-class instance counts, the guideline, and a held-out split fixed before any model is trained.
Need data for Mean Average Precision (mAP)?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.