Imagine training for a marathon using a faulty GPS watch. You might run extra miles, celebrate false progress, and ultimately miss your target. That’s what using a flawed benchmark is like for AI. A new study, “Are We Done With MMLU?” reveals surprising inaccuracies within the popular Massive Multitask Language Understanding (MMLU) benchmark, a key tool for evaluating the capabilities of large language models (LLMs). Researchers discovered a range of errors, from simple parsing mistakes to complex contextual issues. For instance, a shocking 57% of virology questions had flaws, including one suggesting the U.S. army should intervene in West Africa to prevent Ebola outbreaks. This study isn’t just nitpicking; it highlights how these errors can skew our understanding of LLM progress. By creating a refined subset called MMLU-Redux, containing 3,000 manually checked questions, they found that LLMs performed significantly differently, sometimes even changing their ranking compared to the original MMLU. So, can we automatically fix these issues? The team explored techniques using LLMs themselves to detect errors, leveraging techniques like in-context learning and retrieval augmented generation. While there was some progress, the task proved challenging, emphasizing the need for more robust methods. This research serves as a crucial wake-up call. As LLMs become increasingly integrated into our lives, relying on faulty benchmarks can lead to misinterpretations of their true capabilities. The work on MMLU-Redux provides a starting point for creating more reliable evaluation methods, paving the way for a more accurate understanding of AI’s strengths and weaknesses.