Imagine a world where every exam had different questions, or where the same student could get different grades on the same exam depending on the day. Sounds chaotic, right? Well, that's kind of the situation with how we currently evaluate Large Language Models (LLMs). Current benchmarks, the tests we use to measure an LLM's abilities, are often unreliable due to the inherent randomness of these models. Even with identical prompts and settings, an LLM can produce varying outputs, making it hard to get a consistent score. Researchers explored this issue using benchmarks designed to test an LLM's spatial reasoning skills – specifically, their ability to answer questions involving cardinal directions. They found that running the same tests multiple times produced a wide range of scores, creating significant uncertainty about the true performance of the models. They used a statistical tool called a 'prediction interval' to measure this uncertainty, which essentially shows the likely range of scores for future tests. Surprisingly, they found that some LLM scores varied widely, even when supposed to be fixed to a single, deterministic output. So, what does this mean? It highlights a critical need for more robust evaluation methods in the LLM space. Simply running a benchmark once and reporting the score isn't enough. Researchers suggest running tests multiple times and using statistical techniques like prediction intervals to quantify the uncertainty. While setting fixed seeds for generating random numbers can help reduce variability, it's not a foolproof solution. The researchers noticed significant differences in performance even between the same underlying model hosted by different providers. This suggests that the hosting environment plays a vital role in benchmark results and should be meticulously documented. As LLMs become increasingly important, ensuring their fair and reproducible assessment is paramount. This research emphasizes the challenges of relying on single-point benchmark scores and encourages more rigorous evaluation practices to understand the true capabilities and limitations of these powerful models.