Large language models (LLMs) have shown remarkable capabilities in various domains, but how well do they truly understand the nuances of medical information? A new study delves into this question by examining the limitations of LLMs in accurately decomposing clinical facts. Researchers explored how well four popular LLMs—GPT-4o, o1-mini, Gemini 1.5 Flash, and Llama 3 8B—could extract individual facts from various clinical note types, including procedure notes, nursing notes, progress notes, and discharge summaries. They created a new dataset, FactEHR, consisting of over 2,000 real-world clinical notes and their corresponding LLM-generated fact decompositions. The study revealed significant discrepancies in how these models interpret and extract facts. Some LLMs generated over twice as many facts per sentence as others, raising concerns about consistency. Interestingly, while the generated facts were generally accurate when compared back to the original notes, LLMs frequently missed crucial details. This inconsistency poses a significant challenge for building reliable, fact-based medical applications using LLMs. The research also highlights the limitations of current evaluation methods that rely on fact decomposition. The differences in how each LLM breaks down information make it difficult to compare their performance accurately. For example, evaluating models based on the number of 'facts' extracted can be misleading if the 'facts' themselves aren't consistently defined. This research contributes FactEHR, a valuable resource for future research in clinical natural language processing. The dataset includes not only the fact decompositions but also nearly a million entailment pairs—combinations of original text and extracted facts—allowing researchers to test how well LLMs can determine if a fact logically follows from a given piece of text. This work highlights a crucial next step for medical AI: developing more robust methods for LLMs to process and understand the complexities of clinical language, ensuring that AI-powered healthcare tools can reliably and accurately interpret patient information.