Temporal Action Localization
Short answer. Temporal action localization finds where in a video each action starts and ends, so the output is an interval with a label and a confidence rather than a label for the whole clip. It is evaluated by matching predicted intervals to ground truth intervals with temporal intersection over union, then computing average precision. Two things drive the score more than model quality: the overlap threshold, and whether the boundaries were annotated to a consistent rule. A boundary that is 0.6 correct is a hit at one threshold and a miss at another.
How to do it
The unit is an interval, not a clip
A clip classification task labels the whole video, so the annotation is one value and the evaluation is a confusion matrix. A localization task labels a span inside the video, so a two-minute recording may contain three actions, several seconds of transition between them, and long stretches of nothing. The unit count is the number of intervals, not the number of videos.
This changes the annotation cost model the same way it does for object detection: a video with no action still has to be watched to establish that it has none, and a video with many short actions costs far more per minute than one with a single long action. Quote per interval with a separate rate for watching, not per video.
- the label is a start and end time, a class, and usually a confidence
- the unit is the interval, so density varies between videos
- gaps between actions are part of the annotation, not empty space
Temporal intersection over union
The overlap between two intervals is their intersection divided by their union, both measured in frames or seconds. A prediction that covers exactly the right span scores 1.0. A prediction that starts two frames early on a twenty-frame action loses ten percent, and the same two frames on a four-frame action halves the score, so short actions are much less forgiving of the same boundary error.
That asymmetry is why short actions dominate the error analysis. An annotator who is consistently two frames late will produce a dataset where long actions score well and short ones do not, and no amount of model training fixes an annotation rule that does not state where the boundary is. The guideline has to say whether the action starts at the first frame where motion is visible or the frame before it.
Average precision and the threshold it was computed at
Predictions are sorted by confidence and matched to ground truth in that order, each ground truth interval used at most once. A prediction counts as a hit if its overlap with an unmatched ground truth exceeds the threshold. Precision and recall are then computed at every rank, and the area under that curve is the average precision.
The threshold is reported as part of the metric, as mAP at 0.5 or an average over several thresholds. Raising it from 0.5 to 0.7 makes every boundary error count more, so the same predictions produce a much lower number. A result quoted without its threshold cannot be compared with anything, and a supplier quoting an action detection accuracy without one is quoting a number that means nothing.
Code
Match four predicted intervals against three ground truth intervals at two thresholds, build the precision and recall curve, and compute average precision at each.
# a 100 frame video with 3 ground truth action intervals; 4 predicted intervals,
# each carrying a confidence
GT = [(10, 30), (45, 60), (75, 95)]
PRED = [(12, 32, 0.9), (44, 58, 0.8), (70, 90, 0.7), (20, 25, 0.6)]
def tiou(a, b):
inter = max(0, min(a[1], b[1]) - max(a[0], b[0]))
union = (a[1] - a[0]) + (b[1] - b[0]) - inter
return inter / union if union else 0.0
print("prediction -> best matching ground truth")
for s, e, c in PRED:
scores = [(tiou((s, e), g), i) for i, g in enumerate(GT)]
v, i = max(scores)
print(" [%3d, %3d] conf %.1f gt %d [%3d, %3d] tIoU %.3f"
% (s, e, c, i + 1, GT[i][0], GT[i][1], v))
print()
def average_precision(thr):
order = sorted(PRED, key=lambda r: -r[2])
used, rows = set(), []
for s, e, c in order:
best, bi = 0.0, -1
for i, g in enumerate(GT):
v = tiou((s, e), g)
if v > best and i not in used:
best, bi = v, i
if best >= thr and bi >= 0:
used.add(bi)
rows.append((c, True))
else:
rows.append((c, False))
tp = 0
recalls, precs = [], []
for n, (c, hit) in enumerate(rows, 1):
tp += hit
recalls.append(tp / len(GT))
precs.append(tp / n)
env, best = [], 0.0
for p in reversed(precs):
best = max(best, p)
env.append(best)
env.reverse()
ap, prev = 0.0, 0.0
for r, p in zip(recalls, env):
ap += (r - prev) * p
prev = r
return ap, rows, recalls, precs
for thr in (0.5, 0.7):
ap, rows, recalls, precs = average_precision(thr)
hits = sum(1 for _, h in rows if h)
print("tIoU threshold %.2f -> AP %.3f (TP %d, FP %d, FN %d)"
% (thr, ap, hits, len(PRED) - hits, len(GT) - hits))
for i, ((c, h), r, p) in enumerate(zip(rows, recalls, precs), 1):
print(" rank %d conf %.1f %s precision %.3f recall %.3f"
% (i, c, "hit " if h else "miss", p, r))
print()
print("the third prediction overlaps its ground truth by 0.600, so it counts at a")
print("threshold of 0.50 and does not at 0.70, and AP drops from 1.000 to 0.667.")
print("report the threshold. a boundary that is 60 percent correct is not the same")
print("result as one that is 80 percent correct, and only the threshold says which.")- Standard library only. Intervals are in frames; with a frame rate the same code works in seconds.
- Expected: the four predictions overlap their best ground truth at 0.818, 0.812, 0.600 and 0.250. The fourth prediction overlaps the first ground truth, which is already taken, so it can never become a hit.
- Expected: at a threshold of 0.50 all three real actions are found and the extra prediction is the only error, giving AP 1.000 with TP 3, FP 1, FN 0. At 0.70 the third prediction at 0.600 falls out, giving AP 0.667 with TP 2, FP 2, FN 1.
- The precision envelope is computed by scanning the precision values from the end and keeping the running maximum, which is the all-point interpolation used by the detection benchmarks. Without it the curve is not monotone and the area comes out slightly low.
- Set the third prediction to conf 1.0 and rerun. It sorts first, becomes a hit at 0.50 and a miss at 0.70, and the precision curve changes shape at both thresholds, which is the reason confidence calibration matters separately from localization accuracy.
Where this goes wrong
Comparing average precision across different thresholds
An AP at 0.5 and an AP at 0.7 are different metrics, and a pipeline tuned on one will look better on it. When a supplier reports action detection accuracy, the threshold has to come with it, along with whether it is a single threshold or an average over several. Without that, two numbers cannot be placed on the same scale.
Annotating boundaries without a written rule
The start of an action can reasonably be the first frame with visible motion, the last frame before it, or the first frame of the preceding pause. Annotators pick differently unless the guideline says. The disagreement is small on long actions and large on short ones, so it shows up as a class-dependent score cap rather than as an obvious labelling problem.
Ignoring transition regions
Between two actions there is usually a stretch that belongs to neither, and a pipeline that forces every frame into a class will place boundaries at the midpoint of the transition rather than at the action edge. The ground truth has to allow a background class, and the evaluation has to count predictions that fill the gap as false positives.
Pricing the work per video
A per-video price assumes an action density that has not been measured. A video with one long action and a video with twenty short ones cost very different amounts to annotate, and the second is where the money goes. Measure the density on a pilot and price per interval with a separate rate for watching.
When to buy the data instead of building the pipeline
Choose the overlap threshold and write the boundary rule yourself, because both are part of the task definition and the score depends on them. Buy the annotation when the video volume is beyond your team or the action classes need domain knowledge, and specify the boundary rule, the background class and the per-interval price. Ask for the interval density on a pilot before agreeing to a total. We deliver interval annotations with the boundary rule and the background convention documented, plus the density measured on the pilot.
Need data for Temporal Action Localization?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.