Large language models (LLMs) are showing promise in tackling problems involving graphs, which are structures that represent relationships between things. Recent efforts have focused on boosting LLMs' graph reasoning skills through specialized training. However, a new study questions whether these "graph LLMs" truly understand graph reasoning or if they're simply memorizing patterns in the training data. Researchers developed a benchmark called NLGIFT to test how well LLMs generalize their graph reasoning abilities. NLGIFT presents LLMs with various graph problems, changing the wording, numbers, structure, and even the type of reasoning required. The results reveal that while LLMs can handle some shifts in wording or numbers, they struggle when the underlying reasoning or the graph structure changes significantly. For instance, an LLM might succeed in finding the shortest path between two points on a small, simple graph, but fail on a larger, more complex graph. This suggests that LLMs might be latching onto superficial patterns instead of grasping the fundamental principles of graph reasoning. Furthermore, when tested on real-world tasks that involve graph structures, like answering questions that require multiple steps of reasoning, LLMs trained on synthetic graph data showed little to no improvement. This raises concerns about the practical usefulness of current "graph LLM" training methods. Researchers explored several techniques to improve LLMs' ability to generalize graph reasoning, such as mixing code into the training data, using machine-generated explanations, and aligning the LLM's output with human preferences. Early results suggest that aligning LLMs with human feedback is the most promising approach, but much more research is needed. This study highlights a critical challenge in developing truly robust AI systems that can reason with complex structured information. While LLMs have made impressive progress in natural language processing, their ability to reason like humans, especially in domains like graph reasoning, is still a work in progress.