Large language models (LLMs) have wowed us with their ability to write poems, translate languages, and even generate code. But can they truly *understand* complex structures like graphs and execute algorithms on them? New research puts LLMs to the test, exploring their capabilities in tackling classic graph algorithms like Breadth-First Search, Depth-First Search, Dijkstra's algorithm, Floyd-Warshall, and Prim's Minimum Spanning Tree. The results reveal a fascinating gap between LLMs' impressive performance on many tasks and their struggles with multi-step reasoning on explicit graphs. This research introduces MAGMA, a benchmark designed to evaluate how LLMs perform on each stage of these algorithms. Surprisingly, smaller, fine-tuned models often outperform larger foundation models, demonstrating the importance of specialized training. However, even these smaller models show vulnerability to extraneous information, highlighting the delicate balance of providing helpful context without confusing the model. The research delves into specific error types, revealing patterns in how LLMs stumble. For instance, they often hallucinate non-existent nodes or edges, especially when trained only on the input and final output without intermediate steps. This emphasizes the importance of teaching LLMs to break down complex problems step-by-step, much like a human would learn. The implications of this research extend beyond graph algorithms. By understanding LLMs' limitations in structured reasoning, we gain crucial insights for improving their problem-solving abilities across various domains. While LLMs have made tremendous strides, their capacity for true algorithmic reasoning on graphs remains a work in progress. This research underscores the need for smarter prompting techniques and training methods that encourage a deeper understanding of algorithmic logic, paving the way for more sophisticated and reliable AI systems.