Can large language models (LLMs) truly grasp mathematical reasoning, or are they just good at mimicking it? A new study uses combinatorial puzzles to put LLMs like GPT-4 to the test, comparing them not only to other LLMs, but also to human students experienced in math competitions. The results reveal intriguing insights into how LLMs approach problem-solving and where they still fall short. Researchers introduced a novel dataset called Combi-Puzzles, comprising 125 combinatorial problem variations across five distinct categories: common, mathematical, adversarial (with irrelevant information), parameterisation (larger numbers), and linguistic obfuscation (story-based problems). This allowed them to assess not just raw problem-solving ability, but also how well LLMs and humans generalize across different problem representations. GPT-4 emerged as the clear winner among the tested LLMs, even outperforming human participants overall. It particularly shone in problems presented in formal mathematical language, achieving a remarkable 94% accuracy. However, GPT-4's performance wasn't uniform. It struggled with adversarial and linguistically obfuscated problems, suggesting it's sensitive to irrelevant information and narrative distractions. Interestingly, human participants showed a more consistent performance across the different problem variations. While GPT-4 could solve complex problems that stumped humans, it sometimes faltered on simpler ones, especially those wrapped in narratives. This highlights a key difference: humans seem better at grasping the underlying mathematical core of a problem, even when it's presented in an unfamiliar way. This study's findings offer valuable insights into the strengths and limitations of LLM reasoning. While LLMs are clearly powerful tools, they're not yet perfect mathematicians. Their sensitivity to problem phrasing and occasional struggles with basic logic suggest that there's still room for improvement in how they learn and apply mathematical reasoning. The Combi-Puzzles dataset represents a significant step towards more robust and nuanced evaluation of LLM capabilities, paving the way for the development of even more sophisticated and reliable AI problem-solvers.