Imagine walking into a doctor's office, but instead of a human physician, a large language model (LLM) like ChatGPT assesses your symptoms. While this scenario sounds futuristic, LLMs have recently shown impressive results on medical exams. A new study, however, reveals a critical gap between AI's test-taking abilities and true medical understanding. Researchers from Tsinghua University developed a novel evaluation framework called "MultifacetEval" to delve deeper into how well LLMs actually grasp medical knowledge. Instead of just multiple-choice questions, MultifacetEval challenges LLMs with tasks requiring comparison, rectification, discrimination, and verification of medical concepts. Think of it like a doctor needing to not only diagnose an illness from a list of options but also correct a misdiagnosis, distinguish between similar conditions, and validate medical claims. The team tested various general and specialized medical LLMs using two datasets: MultiDiseK, derived from a clinical knowledge base, and MultiMedQA, adapted from a medical licensing exam (MedQA). The results were eye-opening. While some larger LLMs performed well on standard comparison tasks (like multiple-choice medical exams), their proficiency plummeted when facing more complex multifaceted challenges. Smaller LLMs struggled even further, highlighting the importance of model size and training data for robust medical knowledge. LLMs excelled at comparison tasks, likely due to their extensive training on similar question formats in existing benchmarks. However, they faltered in rectification, requiring them to identify and correct medical errors, and in discrimination, demanding nuanced understanding of similar medical concepts. Verification, the ability to confirm or refute medical statements without prompts or clues, proved the most difficult. This research underscores the limitations of using standard tests to assess AI's medical competence. While LLMs can memorize and retrieve information effectively, they currently lack the depth and comprehensiveness of human physicians. Developing future medical AI requires a shift beyond rote learning towards genuine understanding and application of complex medical concepts. The road to AI doctors may be longer than initial benchmarks suggested, but this research illuminates the path forward by highlighting the crucial facets of medical knowledge that demand more attention.