BLEU Score
Short answer. BLEU compares a generated sentence against one or more references by counting matching n-grams, clipping each match to the number of times it appears in the reference, taking the geometric mean of the precisions for orders one through four, and multiplying by a brevity penalty that punishes short output. The clipping step is what stops a system from scoring well by repeating a single common word. BLEU is a corpus metric: a single sentence score is noisy, and the number is only meaningful against another system scored on the same references with the same tokenization.
How to do it
Clipped n-gram precision
For each order from one to four, count how many n-grams in the candidate appear in the reference, but cap each n-gram at the number of times it appears in the reference. Without the cap, a candidate that repeats the word the fifty times would have a unigram precision near one. With the cap, the extra copies count as unmatched, and the precision falls.
The precisions for the four orders are combined with a geometric mean rather than an arithmetic one, so a zero at any order drives the whole score to zero. That is deliberate: a candidate whose word order is scrambled can have perfect unigram precision and no matching bigrams, and the geometric mean is what reports that as a failure rather than as a good score.
The brevity penalty
Precision alone rewards short output, because a candidate of one word that happens to be in the reference has precision one. The brevity penalty multiplies the score by a factor that falls as the candidate gets shorter than the reference, and is exactly one when the candidate is at least as long. The effect is that a system cannot buy precision by saying less.
The penalty is applied to the whole corpus using the total lengths rather than per sentence, so a few very short sentences are absorbed by the rest. This is one reason single-sentence BLEU is not a stable number and why the metric is reported over a test set rather than per example.
Where BLEU stops being informative
BLEU measures surface overlap with one reference, so a correct paraphrase that shares no wording scores near zero. For a task with many valid answers, such as dialogue or summarization, that makes BLEU a weak signal, and it is why those tasks moved to learned metrics. For translation with several references the effect is smaller but not gone.
The number is also tokenization dependent. Word-level BLEU, character-level BLEU and subword BLEU on the same output give different values, and a change of tokenizer between two systems makes the comparison meaningless. Record the tokenizer, the case handling and the number of references alongside any BLEU number you report.
Code
Implement BLEU-4 from scratch, including the clipping step and the brevity penalty, and score four candidates against one reference.
import math
from collections import Counter
def ngrams(tokens, n):
return Counter(tuple(tokens[i:i + n]) for i in range(len(tokens) - n + 1))
def bleu(candidate, reference, max_n=4):
c, r = len(candidate), len(reference)
rows = []
for n in range(1, max_n + 1):
cand = ngrams(candidate, n)
ref = ngrams(reference, n)
clipped = sum(min(cnt, ref[g]) for g, cnt in cand.items())
total = max(sum(cand.values()), 1)
rows.append((n, clipped, total, clipped / total))
if c < r:
bp = math.exp(1.0 - r / c)
else:
bp = 1.0
if any(p == 0.0 for _, _, _, p in rows):
return 0.0, bp, rows
logs = sum(math.log(p) for _, _, _, p in rows) / max_n
return bp * math.exp(logs), bp, rows
ref = "the model was trained on forty hours of read speech".split()
cases = [
("identical", ref),
("dropped two words", "the model trained on forty hours read speech".split()),
("right words, wrong order", "trained speech forty hours model the on read was of".split()),
("half the sentence", "the model was trained on forty hours".split()),
]
for name, cand in cases:
score, bp, rows = bleu(cand, ref)
print("%-26s BLEU-4 %.4f brevity %.3f" % (name, score, bp))
print(" n-gram precision: " + " ".join(
"%d:%d/%d" % (n, cl, tot) for n, cl, tot, _ in rows))
print()
print("the third candidate uses every word of the reference exactly once, so its")
print("unigram precision is 1.000. Only one of its nine bigrams survives, and no")
print("trigram or 4-gram does, so the geometric mean collapses to 0.000")- Standard library only, no numpy needed.
- Expected: identical scores 1.0000. Dropping two words costs a lot, down to 0.3638, because the brevity penalty is 0.779 and the 4-gram precision falls to 1 out of 5. Saying half the sentence gives 0.6514, where every n-gram precision is perfect and the entire loss is the brevity penalty.
- The third row is the one to remember. It has every word of the reference exactly once, so unigram precision is 1.000, and it scores 0.0000 because only one of its nine bigrams appears in the reference and no trigram or 4-gram does.
- Change max_n to 1 and the third candidate scores 1.0000. At max_n of 2 it scores 0.3333, and only from 3 upward does it reach 0.0000. The orders above unigram are the whole reason the metric detects word salad, and the number of orders is a setting you have to state whenever you quote a score.
Where this goes wrong
Comparing BLEU numbers from different tokenizations
Word-level and subword BLEU on the same output differ, and so do cased and uncased variants. Two systems scored with different settings are not comparable, and the gap can exceed the difference the comparison was meant to detect. Fix the tokenizer, the case handling and the reference set before scoring, and record all three with the number.
Reporting a single-sentence BLEU
BLEU is defined over a corpus, and the brevity penalty is applied to total lengths. A per-sentence value swings between zero and one with the choice of one word and carries almost no information. Report the corpus number, and if you need per-example detail, show the n-gram matches rather than a score.
Using BLEU for a task with many valid answers
A correct paraphrase sharing no wording with the reference scores near zero, and a fluent but wrong answer that happens to reuse the reference vocabulary scores higher. For summarization, dialogue, or any open-ended generation, surface overlap is a weak proxy and the metric will mislead you about which system is better. Use it for translation against multiple references, and use something else where the task allows many correct outputs.
Optimizing against BLEU and expecting quality
Systems tuned to maximize BLEU find the surface patterns the metric rewards, which include reusing reference phrases and producing safe, short output that avoids risk. The score rises while the output gets more conservative. If BLEU is a proxy for something a human cares about, keep a human rating on a sample as the actual target and treat BLEU as a fast check.
When to buy the data instead of building the pipeline
Implement the metric yourself; it is forty lines and you need to control the tokenization. Buy the evaluation data when you need references you cannot write: translated or transcribed text in a language and domain your team does not cover, produced by people whose first language it is. Ask how many references per item, who produced them, and whether the same people produced the system output, because a reference written by the system author is not an independent reference. We deliver reference sets with the provenance and the count of references per item documented.
More technical reference
Need data for BLEU Score?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.