Large language models (LLMs) are increasingly used for all sorts of tasks, even evaluating the quality of other AI-generated text. But can we trust their judgment? New research explores whether LLMs like Google's Gemini are truly up to the task of assessing subjective qualities like coherence, consistency, and relevance in generated summaries and dialog. The results reveal a surprising consistency in the LLM's evaluations across different prompting styles. However, a critical vulnerability emerges when the input is slightly tweaked. By introducing subtle 'perturbations' – essentially, small changes that introduce contradictions – researchers discovered that LLMs can be easily misled, leading to drastically different and unreliable evaluations. This susceptibility to manipulation raises serious questions about using LLMs as standalone judges of quality. While they excel in controlled settings, their real-world application for subjective tasks requires a much higher level of robustness. The research also hints at future directions, including the need for more rigorous training and investigation of smaller, more efficient language models that might offer a balance between performance and robustness.