Large Language Models (LLMs) are rapidly changing our world, but can we trust them with something as crucial as medical information? A new research project from the Technical University of Darmstadt put LLMs to the test, evaluating their ability to answer clinical questions accurately and safely. The study, designed as a shared task for students in a Foundations of Language Technology course, focused on identifying harmful or misleading information generated by LLMs in response to complex health queries. Students meticulously annotated LLM-generated answers, comparing them to expert-provided responses from the trusted Cochrane Clinical Answers database. They looked for everything from outright contradictions and exaggerations to subtle understatements that could lead patients astray. The results highlighted the challenges of ensuring LLMs provide safe and reliable health advice. While some LLMs were better than others at sticking to the facts, the study revealed a tendency for these models to generate responses that didn't align with established medical knowledge. The research also offered a valuable learning experience for the students. They gained firsthand experience in data annotation, LLM evaluation, and the broader ethical considerations surrounding AI in healthcare. The hands-on nature of the project allowed them to grapple with real-world implications of LLM technology, preparing them for the complex landscape of AI development and deployment. This research underscores the importance of ongoing efforts to improve the accuracy and trustworthiness of LLM-generated health information. As LLMs become increasingly integrated into our lives, ensuring they provide reliable medical guidance is crucial for patient safety and informed decision-making. The freely available dataset created through this project will be a valuable tool for future research, fostering further investigation into the complex relationship between LLMs and healthcare. It's a vital step towards a future where AI empowers both patients and healthcare professionals with accurate and reliable information.