ROUGE Score
Short answer. ROUGE measures how much of the reference appears in the generated text, which is the opposite direction from BLEU. ROUGE-N counts overlapping n-grams divided by the number of n-grams in the reference, so it is recall oriented. ROUGE-L uses the longest common subsequence, which rewards words that appear in the same order without requiring them to be adjacent. The pair is what makes ROUGE useful: a word salad scores 1.000 on ROUGE-1 and 0.231 on ROUGE-L, and reading both tells you which failure you have.
How to do it
Recall orientation and what it buys
The denominator is the reference, not the candidate, so ROUGE answers how much of the reference the system managed to cover. That is the right question for summarization, where the risk is leaving out a key point rather than adding a wrong one. BLEU asks the opposite question, which is why the two metrics disagree about the same output.
The cost of recall orientation is that longer output is rewarded: adding words that happen to appear in the reference raises the score. ROUGE-1 alone therefore cannot distinguish a good summary from a shuffled list of the reference words, and that is the failure ROUGE-L exists to catch.
ROUGE-L and the longest common subsequence
ROUGE-L finds the longest subsequence of words that appears in both texts in the same order, without requiring the words to be adjacent. It reports recall as that length over the reference length, precision as that length over the candidate length, and an F measure combining them. Because it needs the order to match, a scrambled candidate scores low even though it uses every word.
The F measure is the number to report when the candidate can be either too short or too long. Recall alone rewards padding, precision alone rewards truncation, and the F measure is the balance. Say which of the three you are reporting, because ROUGE-L is used to mean all three in different papers.
Choosing between ROUGE and a learned metric
ROUGE is cheap, deterministic, and explainable, which is why it survives. Its weakness is that it matches surface forms, so a summary that says the same thing in different words scores low, and a summary that copies a sentence from the reference scores high whether or not it captured the point. For abstractive summarization the gap between ROUGE and human judgement is large enough that papers report both.
The practical use is as a fast regression check rather than as the objective. If ROUGE drops after a change, something changed in the output and it is worth looking. If ROUGE rises, that is not evidence that the summary got better, and a sample read by a person is still what decides.
Code
Implement ROUGE-1 and ROUGE-L from scratch, including the longest common subsequence, and score three candidates that fail in different ways.
from collections import Counter
def rouge_n(candidate, reference, n=1):
def grams(t):
return Counter(tuple(t[i:i + n]) for i in range(len(t) - n + 1))
cand, ref = grams(candidate), grams(reference)
overlap = sum(min(cnt, ref[g]) for g, cnt in cand.items())
total = sum(ref.values())
return overlap, total, overlap / total if total else 0.0
def lcs_len(a, b):
prev = [0] * (len(b) + 1)
for x in a:
cur = [0]
for j, y in enumerate(b):
if x == y:
cur.append(prev[j] + 1)
else:
cur.append(max(prev[j + 1], cur[j]))
prev = cur
return prev[-1]
def rouge_l(candidate, reference):
lcs = lcs_len(candidate, reference)
rec = lcs / len(reference) if reference else 0.0
pre = lcs / len(candidate) if candidate else 0.0
f = 2 * pre * rec / (pre + rec) if (pre + rec) else 0.0
return lcs, rec, pre, f
ref = "the annotation guideline says a box covers the visible extent of the object".split()
cands = [
("same order, shorter", "a box covers the visible extent of the object".split()),
("same words, shuffled", "object the of extent visible the covers box a guideline annotation the says".split()),
("one phrase replaced", "the annotation guideline says a box covers the full outline of the object".split()),
]
for name, cand in cands:
ov, tot, r1 = rouge_n(cand, ref, 1)
lcs, rec, pre, f = rouge_l(cand, ref)
print("%-22s ROUGE-1 %.3f (%d/%d) ROUGE-L rec %.3f pre %.3f F %.3f"
% (name, r1, ov, tot, rec, pre, f))
print()
print("ROUGE-1 hands the shuffled candidate a perfect 1.000, because it uses the same")
print("words. ROUGE-L drops it to 0.231 because the longest shared run is three words.")
print("Read both, and treat a high ROUGE-1 with a low ROUGE-L as a sign of word salad.")- Standard library only. The longest common subsequence is computed with the usual dynamic programming table, one row at a time, so memory is linear in the reference length.
- Expected: the shorter candidate scores ROUGE-1 0.692 and ROUGE-L F 0.818, because it drops four reference words but keeps everything it has in order. The shuffled candidate scores ROUGE-1 1.000 and ROUGE-L F 0.231, which is the word salad signature.
- The third candidate replaces one phrase and lands at 0.846 on both, which is what a reasonable paraphrase looks like: a modest drop on both numbers rather than a large gap between them.
- Print the F measure, not just recall. A candidate that appends the whole reference to itself has perfect recall and a precision near 0.5, and recall alone would not show it.
Where this goes wrong
Reporting recall alone
Recall rewards length, so a system that appends reference words to its output improves its score without improving its summary. Report the F measure, or report recall and precision together, and check the length distribution of the output against the reference before believing a recall improvement.
Reading a ROUGE-1 of 1.000 as a good summary
A perfect unigram recall means every reference word appears somewhere in the output, which a shuffled word list also achieves. Pair it with ROUGE-L, and if the two numbers are far apart, the output has the right words in the wrong order. The gap between them is more informative than either number.
Comparing ROUGE across different preprocessing
Stemming, lowercasing, stopword removal and sentence splitting all change the value, and the ROUGE variants in common use differ in exactly these choices. Two papers reporting ROUGE-L may be reporting different metrics. Fix the preprocessing, write it down, and only compare numbers produced the same way.
Using ROUGE as the training objective
Optimizing ROUGE pushes the model toward copying reference phrasing, because exact matches score and paraphrase does not. The score improves while the summaries get more extractive and less useful. Keep a human rating on a sample as the target and treat ROUGE as a fast check that something did not break.
When to buy the data instead of building the pipeline
Write the metric yourself; the implementation is short and the preprocessing has to match whatever you compare against. Buy the evaluation data when you need reference summaries written by people who know the domain: clinical notes, legal filings, technical reports, where a general annotator cannot tell which sentence matters. Ask how many references per document and whether they were written independently, because a single reference makes ROUGE noisy and correlated references inflate it. We deliver reference sets with the provenance and the per-document reference count documented.
Need data for ROUGE Score?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.