Large Language Models (LLMs) are increasingly being used for complex tasks, even in interactive environments like video games. But can they truly plan and strategize? Researchers have introduced Plancraft, a new benchmark dataset based on Minecraft's crafting system, designed to test the planning and decision-making abilities of LLMs. Unlike simple success/fail metrics, Plancraft dives deeper, evaluating the efficiency and quality of an LLM's solutions by comparing them to a handcrafted expert planner. The dataset features varying complexities of crafting recipes, from basic wooden planks to intricate multi-step items. It even includes intentionally unsolvable tasks to see if LLMs can recognize the impossible, a crucial aspect of real-world problem-solving. Early tests using popular models like Llama and GPT reveal that LLMs, while showing promise, still struggle with the intricate planning required by Plancraft. Providing access to external knowledge, like the Minecraft Wiki, through Retrieval Augmented Generation (RAG) significantly improves performance. This suggests that LLMs benefit greatly from readily available information, much like humans consulting instructions. However, the research also identifies a weakness: fine-tuning smaller models, while improving basic task success, hinders their ability to utilize new tools and strategies. This suggests a trade-off between specialized expertise and generalized problem-solving. Interestingly, tests with image-based inputs instead of text descriptions show a significant drop in performance, indicating that today's LLMs struggle to translate visual information into actionable plans in this context. Plancraft opens exciting new avenues for evaluating and improving LLMs, pushing them beyond basic task completion and towards true strategic thinking in dynamic, interactive worlds.