Ever glanced at an AI-generated meeting summary and wondered if it truly captured the essence of the discussion? Turns out, judging the quality of these summaries is a tricky problem. Traditional metrics like ROUGE and BERTScore, which compare the summary to a human-written reference, often fall short. They struggle to capture nuanced errors like misinterpretations or missing crucial decisions. That's why researchers have been turning to Large Language Models (LLMs), like those powering ChatGPT, to act as judges. These LLMs, with their advanced understanding of language, hold the potential to evaluate summaries on a deeper level. But even LLMs aren't perfect. They can be inconsistent, sometimes missing errors or being overly harsh in their judgments. A new research project called MESA tackles these challenges head-on. Imagine a team of LLM experts, each specializing in a specific type of error, like redundancy, incoherence, or factual hallucinations. MESA sets up this virtual team, giving them a three-step process to dissect each summary. First, they identify potential errors. Then, they assess the severity of those errors. Finally, they synthesize their findings into a comprehensive quality score. This multi-LLM approach, combined with a feedback loop that learns from human annotations, helps refine the LLMs' understanding of what makes a good summary. The results? MESA significantly outperforms existing methods, showing a stronger correlation with human judgment in both spotting errors and gauging their impact. While this research primarily focuses on meeting summaries, its implications are far-reaching. The MESA framework offers a new way to evaluate any LLM-generated text, bringing us closer to AI that can truly understand and respond to our needs.