Large language models (LLMs) are making waves in healthcare, promising to revolutionize everything from diagnosis to drug discovery. But are these powerful AI systems ready to handle the sensitive nature of medical information? A new study reveals some alarming shortcomings in the safety and trustworthiness of LLMs when applied to real-world medical scenarios. Researchers put 11 popular LLMs, including big names like GPT-4 and Gemini, through a rigorous test called MedGuard, a benchmark specifically designed to assess the safety of medical AI. The results? Current LLMs, regardless of their safety training, generally flunked most of the tests, especially when compared to the performance of human physicians. The study highlights critical weaknesses in key areas like fairness (ensuring equal treatment for all demographics), privacy (safeguarding confidential patient data), resilience (withstanding adversarial attacks), robustness (handling diverse inputs and errors), and truthfulness (avoiding hallucinations and biases). The findings are a wake-up call. While LLMs show promise in medicine, significant improvements are needed before they can be trusted with sensitive patient data. The study underscores the critical need for robust safety measures and careful human oversight. The research team also found that simply making AI models bigger doesn't guarantee better safety. Surprisingly, domain-specific models, trained specifically on biomedical data, often underperformed compared to general-purpose models. This suggests that fine-tuning for medical knowledge doesn't automatically translate to improved safety. The study emphasizes the need for a more holistic approach to AI development, one that prioritizes safety alongside performance. Prompt engineering, a common technique to improve LLM performance, also proved largely ineffective in boosting safety. This highlights the limitations of quick fixes and emphasizes the need for deeper, more fundamental changes in how these models are trained and designed. The researchers created the MedGuard benchmark to provide a standardized way to evaluate the safety of medical AI. This tool, along with an online leaderboard, will help researchers and developers build safer and more trustworthy AI systems for healthcare, paving the way for a future where AI and humans work together to improve patient care.