Imagine having an AI assistant that could instantly sift through millions of medical records and provide evidence-based answers to complex clinical questions. That's the promise of large language models (LLMs) in healthcare. But how close are we to this reality? A new study puts several LLM-based systems to the test, comparing their ability to answer real-world clinical questions. Researchers evaluated five systems, including general-purpose LLMs like ChatGPT and specialized medical AI platforms like OpenEvidence and ChatRWD. The results reveal a stark difference in performance. General LLMs struggled, often providing irrelevant or even inaccurate information. They hallucinated sources, making up studies that don't exist, a dangerous prospect in a field where decisions have life-or-death consequences. Specialized systems fared better. OpenEvidence, which uses "retrieval augmented generation" to access curated medical literature, provided relevant and evidence-based answers in about a quarter of cases. ChatRWD, an AI agent that generates new studies from real-world data, impressed by accurately answering over half of the questions. Interestingly, ChatRWD truly shone when tackling novel questions, those without existing published research. This highlights a crucial limitation of current medical AI: reliance on past data. General LLMs, trained on historical information, are ill-equipped to handle new or emerging medical scenarios. ChatRWD's ability to generate new studies on demand makes it a powerful tool for addressing these evidence gaps. The study highlights both the potential and the pitfalls of using AI in healthcare. While general-purpose LLMs are not ready for prime time, specialized systems show promise. The future likely lies in combining these approaches – leveraging the summarization power of LLMs with the data-driven insights of AI agents. This synergistic approach could empower doctors with the real-time evidence they need to make the best decisions for their patients.