Logic puzzles can be tricky even for humans, but they present a unique challenge to AI. We tend to combine logical reasoning with our understanding of the world, something that's still difficult for large language models (LLMs). New research explores this intriguing blind spot using a novel dataset called RULEBREAKERS. This dataset contains examples of logical arguments where a conclusion is technically valid according to formal logic rules, but clashes with common sense. For example, consider the argument: 'If Anne is in France, then she is not in Paris. Anne is in Paris. Therefore, Anne is not in France.' Logically, this follows the rule of modus tollens. However, we know Paris is *in* France, so the conclusion is nonsensical. This research uses such rulebreaker examples, along with logically sound arguments, to test how well LLMs can truly *reason*. The results are surprising. While many LLMs struggle to identify the rulebreakers, showing they sometimes over-apply logical rules without considering real-world context, they also demonstrate a hidden ability. By examining the model's confidence levels in its answers, researchers found a latent potential to distinguish between sense and nonsense. This suggests that the seeds of genuine reasoning are present, even if they're not fully developed. This research sheds light on a crucial area of AI development: the integration of formal logic with real-world knowledge. LLMs are powerful tools, but they still have a way to go before they can reason like humans. The RULEBREAKERS challenge provides valuable insights into the current limitations of AI reasoning, paving the way for the development of more robust and reliable language models.