Scaling up inference compute by repeatedly sampling large language models (LLMs) has shown promising results in tasks like mathematical reasoning and code generation. The more attempts an LLM gets, the higher the likelihood of finding a correct solution, right? New research suggests the gains might not be as impressive as they seem. The study “Keep Guessing? When Considering Inference Scaling, Mind the Baselines” argues that current benchmark datasets often have skewed answer distributions, favoring common answers. This means LLMs could be succeeding through lucky guesses rather than genuine reasoning. To test this, researchers introduced a simple baseline: enumerate answers based on their frequency in the training data. Surprisingly, this baseline sometimes outperformed repeated LLM sampling, particularly on tasks with a limited answer set. Even for models that did outperform the baseline, a hybrid approach—combining a few LLM samples with frequent-answer guessing—achieved nearly identical results with far less computational effort. This raises questions about the effectiveness of massive repeated sampling. While scaling inference compute holds promise, this study highlights the importance of carefully chosen datasets and baselines to avoid overestimating LLM capabilities. It seems some LLMs might be getting the right answer for the wrong reasons, highlighting potential pitfalls for techniques that reinforce learning based on solely the final answer's correctness, as they could be rewarding faulty logic. The next step in LLM research involves creating more challenging benchmarks and exploring how LLMs can truly improve their reasoning skills rather than just getting better at guessing.