Large language models (LLMs) have taken the world by storm, generating human-like text and even passing challenging exams. But can they truly *reason*? A new study introduces the Generalized Associative Recall (GAR) benchmark, a clever set of puzzles designed to test the compositional relational reasoning (CRR) skills of LLMs. Think of it as a logic test for AI. CRR is the ability to connect different pieces of information and draw conclusions, a core aspect of human intelligence. The GAR benchmark combines elements of existing tests like associative recall (remembering linked pairs) and knowledge recall (retrieving facts), but with a twist. It introduces varying levels of complexity and even negation to truly challenge these AI giants. The results are surprising. Even top-performing models like GPT-4 struggle with the more complex GAR puzzles, achieving only around 70% accuracy. This reveals a significant 'compositionality gap': LLMs are good at solving individual pieces of the puzzle, but struggle to put them together. Interestingly, bigger models don't always perform better. While scaling generally improves accuracy, the compositionality gap actually *widens* in some models like Llama-2 and Vicuna, suggesting that simply making models larger doesn't automatically improve their reasoning abilities. To understand *why* LLMs struggle, the researchers delved into the inner workings of Vicuna-33B. They discovered specific 'circuits' or pathways within the model responsible for reasoning. Notably, they identified 'True' and 'False' heads – components that light up when processing true or false statements, respectively. These heads play a critical role in how the model judges the truthfulness of information. What’s even more fascinating is that these True/False heads function similarly across different models and even different datasets. This suggests that these components represent a fundamental aspect of how LLMs reason, offering potential avenues for improving their logical capabilities. The GAR benchmark provides valuable insights into the current limitations of LLMs. While these models excel at many tasks, true reasoning remains a challenge. This research emphasizes the need for new approaches, not just bigger models, to bridge the gap between AI and human-like intelligence. The ability to reason is crucial for building truly trustworthy and reliable AI systems, and benchmarks like GAR are essential tools in this ongoing quest.