Can large language models (LLMs) truly understand music? Researchers put them to the test with a fascinating experiment focusing on a core music information task: detecting music entities like song titles and artist names from user-generated content like YouTube titles and Reddit posts. This isn't as simple as it sounds for an AI. User-generated content is messy, full of typos, abbreviations, and informal language. It’s a far cry from the neatly organized data AI models are typically trained on. To make things even more challenging, the researchers built a new dataset, combining existing sources with freshly annotated YouTube video titles. They then benchmarked several leading LLMs, including FireFunction-v2, GPT-4o-mini, Llama3.1-70B, and Mixtral-8x22B, comparing their performance to traditional, fine-tuned smaller language models (SLMs) like BERT and RoBERTa. The LLMs, particularly GPT-4o-mini, outshone the SLMs, demonstrating the power of their vast training data. But the story doesn't end there. The team dug deeper, investigating how these LLMs handled unseen music entities and noisy data. They found a notable link between an LLM's performance and its "exposure" to the entity during its training. In other words, if the LLM had encountered the song or artist before, it was much better at identifying it. This raises interesting questions: are LLMs truly understanding music, or are they simply memorizing patterns from their training data? Furthermore, they explored how robust the LLMs were to errors, like typos and abbreviations, common in user-generated content. Interestingly, they found the type of surrounding text had a big impact. The richer context of a Reddit music request, for example, proved more helpful to the LLM than a concise YouTube video title. While this research demonstrates that LLMs hold great potential for understanding music, it also highlights the critical need for robust evaluation methods, particularly when dealing with the unpredictable landscape of user-generated content. This has implications not just for music information retrieval, but for broader AI applications dealing with real-world data.