BLEU and Its Limits for Evaluating Generated Text

Why BLEU became the default

BLEU exists because comparing generated sentences to human references by hand does not scale. Someone needed a number that could be computed in milliseconds and compared across model checkpoints without ever paying an annotator. BLEU delivers exactly that: a score between zero and one, computed purely from overlap between a candidate sentence and one or more reference sentences, with no learned model and no human in the loop.

The mechanism is simple and that simplicity is the whole appeal. BLEU counts how many n-grams, contiguous sequences of one, two, three, and four words, in the candidate also appear in the reference, then applies a brevity penalty so that short, safe outputs cannot game the score by only producing words they are confident about. The result is a metric that correlates reasonably well with human judgement when comparing similar systems on similar tasks, particularly in the machine translation literature it was built for.

I use BLEU myself when I need a fast, leakage-aware sanity check during development: does this checkpoint look worse than the last one, did the beam search change break something obvious. For that narrow purpose it earns its keep. The trouble starts when people treat it as a proxy for quality rather than a proxy for lexical overlap, because those are not the same thing and the gap between them widens exactly in the cases that matter most.

A worked example of where it breaks

Take a reference sentence: the cat sat quietly on the warm windowsill. Now consider two candidate outputs. Candidate A says the cat sat quietly on the warm windowsill, an exact match, and scores essentially 1.0. Candidate B says a feline rested peacefully on the sunlit ledge, a fluent, accurate paraphrase that any human reader would call correct, but it shares almost no n-grams with the reference and so scores close to zero.

This is not a contrived edge case. It is the everyday reality of language, where the same meaning can be expressed with entirely different words. BLEU has no concept of synonymy, no concept of paraphrase, and no concept of semantic equivalence. It is a string-matching exercise dressed up as a quality metric, and the moment a task rewards diversity of expression, such as summarisation, dialogue, or open-ended generation, that string-matching assumption starts actively punishing good outputs.

The reverse failure is just as revealing. Consider a candidate that says the cat sat quietly on the warm floor, changing only one word, but that word flips the meaning of the sentence in a way a reader would notice. BLEU sees three of four words unchanged and gives a high score anyway, because it has no mechanism for weighting semantically load-bearing words more heavily than filler words. A metric that cannot tell the difference between a harmless paraphrase and a meaning-altering substitution is not measuring what we actually care about.

Reference coverage compounds the problem. BLEU compares against the references it is given, typically one to four per source sentence in standard benchmarks, but any reasonable sentence has dozens of valid phrasings. A system that produces a perfectly fluent, faithful translation using vocabulary the reference writer simply did not choose will be marked down through no fault of its own. In low-resource settings with a single reference per example, this effect is not a rounding error, it can dominate the score.

Where the gap actually bites in practice

The failure mode gets worse as tasks move away from translation. BLEU was designed for a setting where there is a reasonably constrained space of correct outputs and references are plentiful. Summarisation, open-domain dialogue, and creative generation have none of those properties: there are many valid summaries of the same article, many valid replies to the same message, and BLEU's overlap assumption starts measuring something closer to noise than signal. Studies comparing automatic metrics against human ratings in these settings consistently find weak or unstable correlation, which is exactly what the mechanism predicts.

There is also a subtler risk for anyone doing model comparison or leaderboard-style research: BLEU can be gamed, even unintentionally. A model fine-tuned with an objective that rewards n-gram overlap, or a decoding strategy tuned to maximise BLEU on a validation set, will learn to produce safe, high-frequency phrasing that matches references well without necessarily being more useful, more accurate, or more fluent to a human reader. Optimising a metric that only weakly reflects the thing you actually care about is a classic way to make numbers go up while quality stands still or drops.

None of this means BLEU should be discarded. It remains genuinely useful as a fast regression check, as a way to flag catastrophic failures, and as a common reference point when replicating older work that reported BLEU numbers. The mistake is using it as the sole or final word on quality, especially when comparing systems that differ architecturally or when the task allows for genuine diversity in correct answers.

What to do instead

The practical fix is not to find a single metric that replaces BLEU, because no such metric exists. It is to triangulate. Metrics such as ROUGE remain useful for summarisation where recall of key content matters more than exact phrasing. Embedding-based metrics that compare sentence meaning rather than surface tokens catch paraphrases that BLEU misses entirely, though they bring their own biases from whatever model produced the embeddings. None of these are ground truth; they are all approximations with different failure modes.

Human evaluation, done properly with clear rubrics, blind comparisons, and enough annotators to compute inter-rater agreement, is still the most trustworthy signal available, and it is worth budgeting for even in small-scale student or portfolio projects. It does not need to be expensive: a handful of careful pairwise comparisons on a held-out set, with the evaluator blind to which system produced which output, tells you more than another decimal place on a BLEU score.

My own rule of thumb is to treat BLEU, or any single automatic metric, as one column in a table rather than the final verdict. Report it alongside a second automatic metric with different assumptions, a small human evaluation where feasible, and a handful of qualitative examples so a reader can judge for themselves whether the number matches their intuition. If the metric and the examples disagree, trust the examples, and go find out why the metric is lying to you. That habit will save you from shipping a model that scores well and reads badly.