Evaluating how well AI models generate natural language is a tricky business. Traditional methods often fall short, relying on surface-level comparisons that miss the nuances of human language. Recently, large language models (LLMs) have shown promise as automated evaluators, but existing approaches have their own drawbacks: they often rely on expensive, proprietary LLMs like GPT-4 or struggle with nuanced evaluations. A new research paper introduces "Themis," an open-source LLM built specifically for NLG evaluation. What sets Themis apart? Unlike methods chained to reference texts or specific evaluation criteria, Themis offers flexibility and interpretability. It can assess various NLG tasks without needing a reference text and provides detailed explanations for its ratings. The researchers built a massive dataset, "NLG-Eval," comprising half a million samples from nine NLG tasks, including summarization, dialogue generation, and machine translation. This dataset, annotated by both humans and GPT-4, helped train Themis with a unique focus on consistency and alignment with human judgment. Themis not only outperforms existing open-source evaluation models but also rivals proprietary giants like GPT-4 on many tasks. It also demonstrates an ability to generalize to unseen tasks and hold steady even when evaluating noisy or slightly altered texts. Themis offers an exciting path towards more robust and insightful automated NLG evaluation. The open-source nature of the model and dataset promises to empower researchers and developers, accelerating the evolution of natural language generation technology.