Imagine a panel of AI judges, each with unique strengths and weaknesses, working together to determine how relevant search results are to your query. That's the idea behind JudgeBlender, a new framework that uses an ensemble of smaller, open-source AI models to assess the relevance of search results, potentially offering a more efficient and reliable alternative to relying on a single, massive (and expensive) AI model. Traditionally, judging the quality of search results has relied on human assessors, a process that's both costly and time-consuming. Larger language models (LLMs) like GPT-4 have shown promise, but they come with their own limitations, including expense and potential biases. JudgeBlender aims to overcome these challenges by combining the judgments of several smaller AI models, much like a jury reaching a verdict. There are two main flavors of JudgeBlender: PromptBlender, which uses a single LLM with various prompts to elicit different interpretations of relevance, and LLMBlender, which uses multiple distinct LLMs, each with its own specialized prompt. These “judges” offer a broader range of perspectives, aggregating their individual scores to arrive at a final, more robust relevance assessment. Experiments using the LLMJudge benchmark dataset, based on the TREC Deep Learning track, show that JudgeBlender performs competitively with even the most advanced single-model approaches, often showing a stronger correlation with human judgments and an improved ability to rank systems based on their actual performance. Interestingly, the research also suggests that JudgeBlender reduces bias towards systems that use similar underlying AI models, a common problem in current evaluation methods. While the initial findings are encouraging, there's still much to explore. Future work will likely delve into optimizing the mix of models and prompts used in the ensemble, investigating advanced aggregation techniques, and testing JudgeBlender’s performance on a wider variety of datasets and tasks. This research points towards a future where AI-powered search evaluation is not only faster and more cost-effective but also more fair and accurate, ultimately leading to more relevant and satisfying search results for everyone.