Imagine an AI taking – and potentially passing – a rigorous medical licensing exam. Researchers in Japan recently put this idea to the test, using a massive 70-billion parameter large language model (LLM) to tackle the Japanese National Medical Licensing Exam (NMLE). The results offer a fascinating glimpse into how far AI has come in medical reasoning, and what the future might hold for AI in healthcare. The team explored different approaches to fine-tuning the LLM, using a Japanese medical question-and-answer dataset to improve its medical knowledge. They discovered that specializing the LLM for Japanese text significantly boosted its performance on the NMLE, with the model achieving over 50% accuracy. Interestingly, LLMs designed specifically for Japanese text showed a more substantial improvement than models primarily trained on English medical data. This highlighted the importance of language-specific training and the optimization of the AI's "tokenizer" – the component that breaks down language into units the model can understand. The research also revealed the surprising impact of subtle changes in the way questions were phrased (prompting). Even small wording differences could affect the model’s accuracy by up to 8%. This suggests that the way we communicate with AI is crucial for getting accurate and consistent results. While these results are promising, the AI still has ground to cover compared to GPT-4, a state-of-the-art model tested on the same exam in previous studies. The research team acknowledges that current evaluation methods, based on multiple-choice questions, may not fully capture the complexity of medical reasoning. They emphasize the need for larger, more diverse datasets and better ways to measure AI’s clinical knowledge. This study raises important questions about the future of AI in medicine. While an AI doctor might still be a distant prospect, this research shows that large language models, when properly trained, can acquire a significant amount of medical knowledge. As AI technology continues to develop, we can expect further advancements in medical question-answering, with potential applications in medical education, clinical decision support, and patient care.