The mountain of academic papers grows taller every year, a testament to human ingenuity and a looming challenge for researchers tasked with reviewing them. Could large language models (LLMs), adept at generating human-like text, help shoulder this burden? A new study explored this question, delving into whether LLMs can effectively critique and "meta-review" research papers. The results offer a fascinating glimpse into the potential—and limitations—of AI in the world of academic peer review. Researchers created a dataset, ReviewCritique, comprised of NLP papers, human-written reviews, LLM-generated reviews, and expert annotations flagging deficiencies in each. They found that while LLMs can mimic the structure of a review, they frequently miss the mark in terms of substance. LLMs often generated generic criticisms, missed crucial details within the papers, and struggled to distinguish between accepted and rejected submissions. One surprising finding was LLMs' tendency to praise the writing quality of papers even when human reviewers deemed it unclear. Moreover, when tasked with identifying flaws in human-written reviews (meta-reviewing), the LLMs fell short, even the most advanced ones. Their explanations for "deficient" segments often lacked depth and insight, highlighting the crucial role of human expertise in this nuanced task. This research doesn't advocate replacing human reviewers with AI. Instead, it exposes the current inadequacies of LLMs in replicating the critical thinking and domain expertise required for effective peer review. Looking ahead, the challenge lies in refining LLMs to grasp the subtleties of research, provide genuinely constructive feedback, and ultimately, contribute to a more efficient and robust peer review process.