Imagine an AI that could ace any exam, not by cheating, but by deeply understanding the nuances of the specific test. Google researchers have created just that with their “Specialist” method. While large language models (LLMs) like Gemini are impressive generalists, they sometimes falter on specialized evaluations. This research transforms LLMs into laser-focused experts, dramatically boosting their performance in evaluating tasks like machine translation. The team achieved this by feeding the LLM examples of prior human evaluations for the *same* test questions, akin to giving a student targeted practice exams. This allows the LLM to learn the subtle preferences and criteria of individual human evaluators, much like a student learns the grading style of a particular teacher. The results are astonishing. Their specialized AI, called Specialist AutoMQM, outperformed the current state-of-the-art, XCOMET, by a whopping 54% in identifying errors in translated text. This approach goes beyond mere memorization. The AI not only learns which errors to flag but also understands when to abstain from marking something as incorrect, demonstrating a deeper grasp of the evaluation criteria. Surprisingly, the research also reveals that these specialized AIs essentially 'impersonate' human evaluators, mirroring their individual grading quirks. This reinforces the idea that evaluations are not entirely objective, even among experts. This innovative method has far-reaching implications. Imagine faster, more accurate, and consistent evaluation for a wide range of AI tasks. While challenges remain, like the need for pseudo-SxS ratings (where the same rater evaluates all translations of a given source), Google’s “Specialist” method opens doors to a new era of AI assessment, where “jacks of all trades” become true “masters of one.”