Evaluating the effectiveness of AI, especially generative models, often relies on comparing machine-generated results with human judgments. Traditionally, researchers use correlation metrics to measure how well automatic evaluations align with human assessments. However, a new research paper, "Beyond Correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge," reveals that these correlation scores can be misleading due to inherent uncertainties in human judgment. The study highlights how overlooking variations in human responses can create a false impression of AI's effectiveness. Specifically, when human agreement is low on a task, an AI's judgment might appear comparable or even superior to human consensus simply because human responses are all over the map. This illusion fades as human agreement strengthens, revealing a clearer gap between human and machine performance. To address these issues, the researchers propose three key innovations. First, they suggest stratifying evaluation results based on the level of human agreement. This allows for a more nuanced understanding of where AI excels and where it falls short, relative to the confidence of human judges. Second, for inherently subjective tasks where variation in human perception is expected, the researchers introduce a new metric called "binned Jensen-Shannon Divergence for perception" (JSb). This metric compares the distributions of human and machine judgments, acknowledging the range of acceptable human responses rather than relying on a single "correct" answer. Finally, the paper introduces "perception charts" to visually represent the differences and similarities between human and machine evaluations. These charts provide a much more intuitive understanding of how AI aligns with human perception across different categories. These innovations provide a more robust framework for evaluating AI and LLMs, particularly in subjective domains. By acknowledging human uncertainty, we can gain a more accurate and nuanced picture of how well AI truly approximates human judgment.