Imagine asking your AI assistant for health advice. Would you trust it? With large language models (LLMs) becoming increasingly integrated into our lives, their ability to handle health-related inquiries is under scrutiny. A new research paper introduces CHBench, a groundbreaking benchmark designed to evaluate the performance of Chinese LLMs in providing accurate and safe health information. This benchmark focuses on both physical and mental health, covering a wide range of topics from everyday wellness to complex medical scenarios. Researchers used real-world questions sourced from online forums, exams, and existing datasets to test the models. They crafted CHBench specifically to highlight the challenges LLMs face with nuanced queries, including Chinese idioms and medical terminology. Interestingly, they used another LLM, ERNIE Bot, as a benchmark for generating gold-standard responses. The results? While current LLMs show some understanding of health topics, the study reveals significant room for improvement. Many models struggled with accuracy, sometimes giving misaligned or even harmful advice. This highlights a crucial need for further research before we can rely on AI for medical guidance. CHBench marks a significant step towards building more reliable and trustworthy health-focused LLMs, paving the way for a future where AI can play a more helpful and informed role in healthcare.