Do Bigger Evaluation Datasets Make Your Results More Significant?
The size of the test set shouldn’t have any impact on the evaluation, provided that the test set has been correctly created. Increasing its size shouldn’t change the p-value of…
Scientific Credibility in Machine Translation Research: Pitfalls and Promising Trends
Are we at a turning point? My conclusions from the annotation of 1,000+ scientific papers
BLEU: A Misunderstood Metric from Another Age
In this article, we will go back 20 years ago to expose the main reasons that brought BLEU to existence and made it a very successful metric. We will look…
Why the Evaluation of OpenAI Whisper Is Not Entirely Credible
Whisper is evaluated on 6 tasks (section 3 of the research paper). I demonstrate that the conclusions drawn from 3 of these evaluation tasks are flawed ❌ or misleading ❌.