Large Language Models (LLMs) have shown remarkable progress in answering complex questions, but how do they fare when reasoning requires navigating interconnected information like a map? A new research paper introduces the Graph Reasoning-Structured Question Answering Dataset (GRS-QA), a novel approach to testing AI’s ability to handle multi-hop reasoning. Instead of simply providing supporting text, GRS-QA presents questions alongside “reasoning graphs.” These graphs represent the logical connections between pieces of information, much like a map connects locations. This allows researchers to see not just *if* an LLM gets the right answer, but *how* it arrives at its conclusion. The researchers built GRS-QA using existing question-answering datasets, transforming supporting facts into graph structures where nodes represent sentences and edges represent logical relationships. They then tested several LLMs (Llama3, GPT-3.5, and GPT4o-mini) to see how well they could answer questions using these graphs. They also experimented with different ways of presenting information, including using retrieval methods like BM25, providing unstructured text, and even introducing “negative” reasoning graphs with incorrect connections to throw the LLMs off track. The results revealed that LLMs perform differently depending on the complexity of the reasoning graph. While they excel at simpler graphs, performance drops significantly as the connections become more intricate. Interestingly, providing the LLMs with the correct reasoning graph often improved their performance, suggesting that explicit logical pathways can be beneficial. However, introducing incorrect graphs led to decreased performance, highlighting the LLMs’ vulnerability to misleading information. GRS-QA provides a valuable new tool for understanding the strengths and weaknesses of LLMs in multi-hop reasoning. It suggests that while LLMs have made impressive strides, they still struggle with complex logical structures. Future research could explore how to help LLMs better understand and utilize graph-based information, potentially leading to more robust and reliable AI reasoning capabilities. One limitation of the current dataset is the uneven distribution of graph types, with simpler structures overrepresented. Future work could address this by generating synthetic data to balance the representation of complex reasoning patterns. Additionally, exploring how to segment the dataset by domain and developing domain-adapted models could further enhance the dataset's utility in evaluating and improving domain-specific reasoning in LLMs. Finally, expanding the types of negative reasoning graphs could yield deeper insights into how LLMs handle complex logical structures and identify specific areas for improvement.