We all know language can be tricky. "Traffic light" versus "light traffic"—same words, totally different meanings. This difference, rooted in something called phrasal semantics, is easy for humans to grasp but poses a significant challenge for AI. A new study from Salesforce Research dives deep into whether today’s powerful large language models (LLMs) truly understand the nuances of phrases. The researchers tested LLMs using three clever datasets, evaluating how well these models understand and reason about the meaning of phrases like "traffic light" compared to similar but distinct phrases. The results? LLMs outperformed older methods, demonstrating a promising ability to grasp these subtle semantic distinctions. However, they’re not perfect. Interestingly, some fancy prompting tricks, like giving the LLM a few examples or asking it to "think step-by-step," didn't always help and sometimes even made things worse. Digging into these errors revealed that LLMs sometimes struggle to identify the most relevant phrase, even when they understand the words themselves. They also get tripped up by unknown concepts (we can’t all be experts on magic eye technology!) and phrases with multiple meanings (like the ever-ambiguous “small beer”). This research reminds us that while LLMs are remarkably adept at understanding and generating language, there's still a gap between how AI and humans process meaning. The next step? Researchers are focusing on improving LLMs’ ability to follow instructions precisely, developing smarter prompts to boost their reasoning skills, and incorporating external knowledge to tackle ambiguous phrases. These advances are key to bridging the gap between human understanding and artificial intelligence, paving the way for even more powerful and context-aware AI systems in the future.