Large language models (LLMs) have shown remarkable capabilities, but can they truly understand relevance like humans do? A new study challenges the assumption that LLMs can replace human judgment in information retrieval. While LLMs have demonstrated a strong correlation with human assessments in some cases, researchers at the University of Waterloo and the University of New Hampshire argue that this correlation is misleading. Their findings highlight discrepancies between LLM and human judgments, especially when evaluating top-performing retrieval systems. One particularly striking example is a system that deliberately gamed the LLM-based evaluation to achieve a high ranking, while performing significantly worse in human evaluations. This raises concerns about the vulnerability of LLM-based assessments to manipulation. The researchers also point out that LLM-based relevance assessment is essentially another form of re-ranking, making it susceptible to biases that favor LLM-generated content. This 'narcissism' of LLMs, coupled with their susceptibility to prompt engineering, calls into question their objectivity. As information retrieval systems increasingly incorporate LLM-based components, the risk of a disconnect between automated evaluations and genuine human needs becomes more significant. The researchers warn that relying solely on automatic judgments could lead to a future where LLMs primarily assess their own assessments, creating a circular and potentially flawed evaluation loop. The debate over the role of LLMs in relevance assessment underscores the ongoing tension between automated efficiency and the irreplaceable value of human understanding in a field designed to serve human needs.