Large language models (LLMs) are getting impressively good at summarizing text, but how do we really know if they're doing a good job? Existing methods for evaluating summaries have some shortcomings. They often focus on narrow aspects like just checking for factual accuracy, use limited datasets, and rely on coarse-grained scoring methods that don’t capture the nuances of a good summary. Researchers have introduced a new benchmark called UniSumEval to address these limitations. UniSumEval takes a more holistic approach by considering a wider range of inputs—different text types, lengths, and topic domains (from news to science fiction, booking dialogues, and more). It provides fine-grained, multi-dimensional annotations that look at not just factual accuracy (faithfulness), but also whether the summary captures all the important information (completeness) and does so concisely (conciseness). Interestingly, the researchers used AI to help create and validate this new benchmark. They used LLMs to help spot potential issues in input texts and to assist human annotators with the complex task of evaluating summaries. Using UniSumEval, they tested nine different LLMs, finding that while proprietary models like GPT-4 generally perform better, the best model varied depending on the specific type of text being summarized. The research also shows that current automatic summary evaluation methods still have room for improvement, particularly when it comes to evaluating conciseness. This new benchmark provides a more robust and comprehensive way to assess LLM-generated summaries and could lead to better automatic evaluation tools in the future, helping us push the boundaries of what's possible with AI summarization.