Large language models (LLMs) have shown promise in various fields, including medicine. But how well do they truly grasp complex medical knowledge? A new study from Tsinghua University introduces PretexEval, a groundbreaking framework designed to thoroughly test LLMs' medical expertise. Instead of relying on static medical benchmarks, which can become outdated or even leaked, PretexEval dynamically generates test questions from existing medical knowledge bases like MedLAMA and DiseK. These knowledge bases contain crucial information for diagnosis and treatment, making them ideal testing grounds for LLMs. The researchers found a surprising gap between LLMs' performance on standard benchmarks and their ability to answer PretexEval's dynamic questions. Why the difference? PretexEval uses a clever technique called 'predicate equivalence transformation.' This creates multiple, diverse versions of the same medical fact, challenging LLMs to demonstrate a deeper understanding beyond simple memorization. Think of it like asking a medical student to explain a concept in multiple ways – it reveals a more nuanced understanding. The results? Even top-performing LLMs like GPT-4 struggled with PretexEval's rigorous testing. While they excelled at standard benchmarks, they faltered when presented with the same information phrased differently. This inconsistency raises concerns about their reliability in real-world medical applications. The study highlights a critical need for LLMs to develop a more comprehensive and consistent grasp of medical knowledge. This research offers valuable insights for the future of medical AI. Improving how LLMs learn and are tested is essential for their safe and effective integration into healthcare. The Tsinghua team's work paves the way for more robust and reliable medical LLMs, ultimately bringing us closer to a future where AI can truly assist healthcare professionals.