Can today’s powerful AI truly grasp the meaning of language, or are they just masters of clever mimicry? A fascinating new research paper delves into this question by exploring how well Large Language Models (LLMs) can extract the core semantic structure of sentences—the underlying logic that connects words and concepts. The researchers focused on a fundamental aspect of meaning: predicate argument structure. This structure dictates how different parts of a sentence relate to each other, ensuring, for instance, that we understand 'the red car hit the blue truck' correctly and don’t mistake which vehicle is which color. To test LLMs' ability to understand these relationships, the researchers tried two main approaches. The first was a question-answering task, where the LLM had to answer simple yes/no questions about the properties of objects within a sentence. While LLMs showed some initial promise, their performance quickly declined when faced with more complex sentences, revealing a potential struggle with truly grasping underlying semantic relationships. The second approach involved training LLMs to translate sentences into formal logical representations, similar to how linguists analyze meaning. Interestingly, this method revealed a tendency of LLMs to ‘hallucinate’—inventing extra details or misrepresenting the relationships between words. While the larger LLMs demonstrated a better ability to generalize to more complex scenarios, the smaller models often overfit the training data, highlighting the ongoing challenge of building robust and reliable AI systems. This research offers valuable insights into the inner workings of LLMs, shedding light on their strengths and limitations in comprehending language. The study also underscores the importance of developing new methods to mitigate hallucinations and improve the consistency with which AI interprets language. The ability of AI to truly understand meaning is not just an abstract academic pursuit, but a critical step towards creating genuinely intelligent systems that can communicate, reason, and interact with us in a meaningful way.