Large language models (LLMs) have shown impressive abilities in various domains, but can they truly master complex, dynamic environments like video games? Researchers are exploring this question with BALROG, a new benchmark designed to test the agentic capabilities of LLMs and vision-language models (VLMs) in a range of game environments. These games, from simple grid worlds to the notoriously difficult NetHack, demand skills like long-term planning, spatial reasoning, and the ability to learn game mechanics. The initial results are intriguing. While top LLMs like GPT-4 show some promise in simpler games, they struggle significantly with more challenging ones, revealing limitations in spatial awareness, systematic exploration, and planning. A surprising finding is that adding visual input often hinders performance, suggesting current VLMs grapple with translating visual information into effective actions. One curious phenomenon observed is the “knowing-doing” gap. LLMs might correctly answer questions about game mechanics but fail to apply this knowledge during gameplay. For example, a model might acknowledge the danger of eating rotten food in NetHack but still consume it, highlighting a disconnect between knowledge and action. BALROG offers not just a benchmark, but also a platform for exploring new ways to enhance LLM performance in long-horizon decision-making. Research directions include in-context learning, advanced reasoning strategies, and improving how VLMs handle visual information. The ultimate goal is to develop agents capable of complex reasoning and adaptation in dynamic environments, a key step toward creating truly general-purpose AI. The quest to conquer the gaming world is pushing the boundaries of AI research and revealing important insights into the nature of intelligence itself.