Imagine an AI that could predict your likelihood of developing a disease based on your medical history. Large Language Models (LLMs), the tech behind tools like ChatGPT, are being explored for just that. But a new study reveals a surprising truth about their medical predictions. While LLMs excel at generating human-like text, accurately estimating the *probability* of a medical outcome is proving tricky. Researchers examined several advanced LLMs, testing them on medical datasets covering various conditions and demographics. They discovered that the straightforward approach of simply asking an LLM for a probability score isn't the most reliable method. A more indirect method, based on analyzing how the LLM arrives at its prediction, proved more accurate. This difference was even more pronounced with smaller LLMs and datasets with uneven representation of different conditions. Why the discrepancy? It turns out that LLMs, despite their impressive linguistic skills, sometimes struggle with the nuances of numerical reasoning. They might be good at identifying patterns but less adept at quantifying uncertainty. This raises a critical question: how much can we trust AI's confidence in its medical predictions? The study highlights the importance of carefully evaluating LLM-generated probabilities before using them in real-world clinical settings. As AI plays an increasingly important role in healthcare, understanding its limitations, especially regarding uncertainty, is crucial. Future research needs to develop better methods for LLMs to express how sure they are about their medical predictions. This could help doctors make more informed decisions and ensure that patients receive the most appropriate care.