We argue that the machine translation community is overly reliant on the Bleu machine translation evaluation metric. We show that an improved Bleu score is neither necessary nor sufficient for achieving an actual improvement in translation quality, and give two significant counterexamples to Bleu’s correlation with human judgments of quality. This offers new potential for research which was previously deemed unpromising by an inability to improve upon Bleu scores.
I like the conclusion which says that we basically have to be careful to figure out what are appropriate vs inappropriate uses of BLEU. If you want to compare blemishes between apples, fine, use BLEU. If you're talking apples vs. oranges, BLEU is not the right tool for the job.
