Large language models (LLMs) like ChatGPT have taken the world by storm, demonstrating impressive abilities in writing, translation, and even coding. But can these AI giants truly *reason*? A new study puts open-source LLMs from the Llama 2 family to the test, challenging them with symbolic mathematical reasoning tasks to probe the limits of their intelligence. Researchers explored whether LLMs can not only solve equations but also understand the underlying logic and extrapolate to more complex problems. They evaluated three versions of Llama 2: a general-purpose chat model and two specialized math whizzes, MetaMath and MAmmoTH. The results show that while larger models do exhibit stronger symbolic reasoning, even the fine-tuned math specialists falter with increasing problem complexity. This suggests that the impressive skills of current LLMs might not reflect a deep grasp of symbolic reasoning like humans possess. Instead, they may rely on pattern recognition and memorization. While larger models and specialized training showed improvements, the study highlights the persistent gap in LLMs' ability to generalize their knowledge to truly novel or complex symbolic scenarios. This research sheds light on the current limitations of AI reasoning, paving the way for future architectural improvements that might unlock a deeper, more human-like form of intelligence in machines.