The race to build bigger, better language models is on. But there’s a dirty secret lurking in the benchmarks we use to measure their progress: contamination. Imagine training a student for a test using questions from the exam itself. They might ace the test, but would they truly understand the subject? The same problem plagues LLMs. New research introduces PaCoST, a clever method to detect benchmark contamination in large language models. PaCoST works by testing how confident an LLM is in answering a question and comparing it to its confidence in answering a rephrased version. If the LLM is significantly more confident about the original question, it suggests the question might have been part of its training data. The startling discovery? Almost *every* popular LLM and benchmark showed signs of contamination! This means many LLM leaderboards might be misleading. This poses a serious problem for LLM evaluation. How can we know if a model is genuinely intelligent or if it’s simply regurgitating training data? The researchers call for new evaluation methods—perhaps dynamic, user-generated benchmarks or tests based on real-world interaction data—to give us a clearer picture of LLM capabilities. The future of AI depends on rigorous, unbiased evaluation. PaCoST is a critical step towards ensuring LLMs are truly learning, not just memorizing.