Imagine an AI judging a competition. Sounds futuristic, right? Large Language Models (LLMs) are increasingly being used to evaluate everything from essays to code, acting as automated judges. But there’s a catch: these AI judges can be biased. They might unfairly favor an answer simply because of its placement (e.g., first or last) or because of seemingly arbitrary ID tokens (e.g., A or B). This “selection bias” undermines the fairness and effectiveness of AI evaluations. New research introduces “CalibraEval,” a clever technique to combat this bias. Instead of trying to explicitly identify and remove the bias, CalibraEval recalibrates the AI’s judgments. It works by transforming the AI’s initial, potentially biased, prediction into a more unbiased one. This transformation is learned by observing the AI's behavior when answer placements and IDs are shuffled. The beauty of CalibraEval is that it doesn't need any labeled data (telling the AI what the “right” answer is). This label-free approach makes it much more practical and scalable. Experiments show that CalibraEval consistently reduces selection bias across various LLMs and evaluation tasks, even boosting overall accuracy! While more research is needed to tackle other types of AI bias, CalibraEval represents a significant step toward building more robust and trustworthy AI judges. This could revolutionize how we evaluate everything from student essays to complex coding challenges, ensuring a fairer and more objective assessment process.