The race to the top of AI leaderboards is on. But what if those leaderboards are flawed? New research suggests that the benchmarks used to evaluate large language models (LLMs) are increasingly vulnerable to manipulation, creating a distorted view of their true capabilities. While LLMs achieve impressive scores on standardized tests, they often struggle with real-world tasks requiring genuine understanding. This discrepancy raises serious questions about the validity of current evaluation methods. Researchers have identified several ways LLM benchmarks can be “hacked.” Overfitting, or excessive optimization for specific tests, leads to inflated scores without genuine improvement in general abilities. Publicly available datasets allow models to memorize answers, while test set contamination inadvertently exposes models to test data during training. Furthermore, developers can exploit the structure of benchmarks through task-specific optimization, achieving high scores without enhancing overall language comprehension. Even adversarial benchmarking, designed to test robustness against malicious inputs, is susceptible to manipulation. The use of human judges introduces further complexities, as human evaluations are prone to inconsistencies and biases related to factors like output formatting and tone. LLMs can learn to exploit these biases, generating outputs that appear convincing to humans even if the content is factually incorrect. Similarly, using other LLMs as judges creates opportunities for model-to-model collusion and the perpetuation of existing biases. These findings have significant implications for how we interpret LLM benchmarks. As models achieve near-perfect scores, we must view these results with skepticism, recognizing the potential for manipulation. No single evaluation method can provide a complete picture of LLM performance. A combination of approaches, along with detailed documentation of methodologies and limitations, is necessary for robust evaluation. Ultimately, greater emphasis should be placed on evaluating LLMs in real-world applications, where the complexity and unpredictability of tasks can reveal their true capabilities. While benchmarks remain valuable, a more critical and nuanced approach to their interpretation is essential as we continue to develop and assess the next generation of AI.