Large language models (LLMs) are making waves in healthcare, but can they truly understand the intricate web of medical knowledge? Researchers recently put LLMs like GPT-4 and specialized medical models to the test, using a novel approach that goes beyond traditional question-answering. They challenged these AI models to build knowledge graphs – visual representations connecting medical concepts and their causes and effects. Think of it as mapping how the AI “thinks” about diseases. A panel of medical students then evaluated these AI-generated graphs for accuracy and completeness. Surprisingly, the specialized medical models weren’t always the top performers. While they tended to be more specific, they sometimes struggled to differentiate between direct causes and contributing factors. Generalist models like GPT-4, on the other hand, demonstrated a broader understanding, accurately capturing complex relationships. This study highlights the importance of evaluating LLMs in new ways, revealing both their strengths and limitations when it comes to truly understanding medical reasoning. It also points to exciting future directions, such as using AI-generated graphs to improve medical knowledge bases and enhance the way LLMs tackle complex clinical problems. The journey to building trustworthy AI doctors is far from over, but this research offers a valuable glimpse into what the future might hold.