Imagine a world where AI could automatically track the best-performing models across every field. That's the promise of leaderboard extraction, a task that automatically identifies and ranks top AI models from research papers. This exciting area is rapidly evolving, and a recent study dives deep into how the latest Large Language Models (LLMs) are tackling this challenge. Researchers explored how different LLMs, like Mistral 7B, Llama-2, GPT-4-Turbo, and GPT-4, perform at extracting key information like the task, dataset, metric, and score from AI research papers. Instead of feeding the entire paper to the LLMs, they experimented with different parts of the text as input. They tried using just the title, abstract, experimental setup, and tables (DocTAET), or only the results, experiments, and conclusions (DocREC), or even the entire paper (DocFULL). Surprisingly, they found that open-source models, especially Mistral 7B, could often outperform the more well-known GPT models. This is especially true when using the DocTAET approach, where less is more! While longer inputs might seem better, this research highlights the power of focused context. Giving the LLM too much information can actually distract it and lead to less accurate leaderboard extraction. For tasks requiring high precision, focusing on results and conclusions within the DocREC approach works well. This research has big implications for the future. Imagine automatically generated leaderboards, updated in real-time, summarizing the best AI models in any given field. While the current research shows promise, there’s still room for growth. Hybrid input methods, tailoring the context to specific domains, and integrating structured data are promising directions. As LLMs continue to evolve, unlocking their potential to extract meaningful insights from research papers will be a game-changer.