Imagine an AI taking – and passing – your medical licensing exam. While that might sound like science fiction, large language models (LLMs) are making strides in complex fields like medicine. Researchers recently put several state-of-the-art LLMs, including specialized medical AI and general-purpose models like GPT-4, to the test using a massive dataset of real Polish medical and dental licensing and specialization exams. The results are surprising. GPT-4 performed remarkably well, approaching human-level proficiency and even surpassing average student scores on some exams. While other LLMs lagged behind, even general-purpose models outperformed specialized medical AIs, likely due to the prevalence of English-language medical data. However, AI’s success wasn't uniform. LLMs struggled with dentistry, particularly orthodontics, revealing gaps in their understanding of nuanced medical specializations. Interestingly, AI excelled in areas like laboratory diagnostics, suggesting that these tasks align well with their data analysis capabilities. The research also explored cross-lingual knowledge transfer. Unsurprisingly, models trained primarily on English data performed better on English translations of the exams. However, as models improve, this performance gap narrows. This study underscores that while AI can achieve impressive results on standardized tests, these scores only reflect a sliver of medical expertise. The complex, human-centered aspects of medical practice, requiring real-world experience, adaptability, and emotional intelligence, remain beyond the scope of current AI capabilities. Passing an exam is one thing; practicing medicine is another entirely. The future implications of LLM integration into healthcare are significant, offering potential tools to support medical professionals. However, responsible development and deployment are crucial, emphasizing human oversight and ethical considerations. These powerful tools must be guided by human expertise to ensure patient safety and responsible use.