Imagine trying to keep up with the rapidly evolving world of Artificial Intelligence research. It's a whirlwind of new models, datasets, and metrics, and the traditional tools for tracking progress are struggling. This is where AI leaderboards come in. These leaderboards rank models based on their performance, providing a much-needed snapshot of the state-of-the-art. But manually curating these leaderboards is a tedious task. What if we could automate it using Large Language Models (LLMs)? A new research paper explores exactly that, investigating how to effectively select the right context from research papers to guide LLMs in automatically extracting information for leaderboard generation. The researchers looked at three different ways of feeding information to the LLM: using the entire paper (DocFULL), selected sections like results and conclusions (DocREC), or a more targeted approach with the title, abstract, and experimental setup (DocTAET). Surprisingly, they found that less is more. While you might think giving the LLM the entire paper would lead to the best results, it actually performed the worst. The DocFULL context gave the LLM too much to sift through, leading to errors and made-up information. The most effective approach was DocTAET, which narrowly focused on the most relevant parts of the paper. This more targeted approach resulted in the most accurate leaderboards, suggesting that filtering out the noise is key for these complex tasks. Both LLMs tested, Mistral 7B and Llama 2 7B, showed significant improvement using this focused context, especially when tested on tasks they hadn't seen before. This is a big win for automation in research, highlighting how careful context selection can lead to more reliable AI systems for extracting crucial information. The research opens up exciting possibilities for keeping up with the rapid pace of AI research and automatically building reliable leaderboards. It also sheds light on how LLMs process information, emphasizing the importance of context in guiding their output. While the automatic extraction of nuanced performance metrics remains a challenge, this work represents a significant step forward in using LLMs to summarize and organize complex scientific information.