Imagine asking your AI assistant not just for facts, but to devise a step-by-step strategy for achieving a complex goal. While recent large language models (LLMs) excel at generating text, true planning remains a significant hurdle. A new benchmark called PlanBench is putting these AI systems to the test, revealing their limitations when it comes to devising plans. Researchers have used block-stacking puzzles as a simple yet effective planning challenge. Even seemingly straightforward scenarios, like rearranging colored blocks on a table, require a level of strategic thinking that many LLMs struggle to grasp. OpenAI's latest model, known as "o1" or "Strawberry", stands apart. Touted as a "Large Reasoning Model" (LRM), it claims to move beyond the retrieval-based approach of previous LLMs and demonstrates a degree of genuine reasoning ability. So, how does o1 fare on PlanBench? Significantly better than its LLM predecessors. It can solve almost all of the simple block-stacking problems in the benchmark. However, the research reveals some intriguing nuances. When given more complex puzzles, o1's performance drops dramatically. Moreover, these more advanced LRMs raise new questions about efficiency and cost. Classical planning systems like Fast Downward can achieve 100% accuracy on the same puzzles, while o1, despite some promising results, is still considerably more expensive and resource-intensive. This highlights the fact that while LRMs like o1 demonstrate a fascinating step toward genuine AI planning, they are still a long way off in providing accuracy guarantees, and efficiency when compared to other solutions. The research suggests that future AI models might benefit from combining LLM creativity with the logical rigor of more traditional planning algorithms, ensuring that the AI can reliably get the job done while providing correctness guarantees and being cost effective.