Imagine having an AI assistant that could automatically fix bugs and implement new features in your codebase. That's the tantalizing promise of code-editing Large Language Models (LLMs). But how good are these LLMs in real-world scenarios, dealing with the complexities of large code repositories? Researchers have developed a new benchmark called RES-Q, designed to put these code-editing LLMs through their paces. Unlike traditional benchmarks that focus on single-file edits, RES-Q presents LLMs with 100 realistic tasks based on actual GitHub commits, requiring them to navigate and modify entire repositories. These tasks aren't simple find-and-replace exercises either. They involve interpreting vague instructions, identifying the relevant files to change, and making complex modifications spanning multiple lines of code. The results are surprising. While closed-source models like Claude 3.5 Sonnet show promising results, even outperforming GPT-4 on some tasks, there's still a significant gap between AI and human developers. Interestingly, limiting the AI's access to the codebase actually *improved* performance for some open-source models, suggesting they struggle to process vast amounts of information effectively. The RES-Q benchmark highlights the challenges of building truly robust code-editing AI. While LLMs can automate certain coding tasks, they still have a long way to go before they replace human developers. The next step? Researchers are focusing on improving how LLMs understand context, reason about code, and handle the ambiguity inherent in software development.