Designing computer chips is a complex process, traditionally relying heavily on human expertise. But what if AI could lend a hand? Researchers are exploring the potential of Large Language Models (LLMs), like the ones powering ChatGPT, to automate parts of chip design, specifically at the Register Transfer Level (RTL) where hardware is described in code. However, evaluating the effectiveness of LLMs for this specialized task has been challenging. Existing benchmarks haven't captured the complexity of real-world chip design projects, which often involve large, interconnected codebases. Enter RTL-Repo, a new benchmark designed to put LLMs through their paces in a more realistic setting. RTL-Repo consists of over 4,000 code samples taken from real-world projects on GitHub, providing a diverse and challenging testing ground. The benchmark tests the ability of LLMs to generate Verilog, a hardware description language, within the context of a large project, not just isolated modules. Initial results show promise, with models like GPT-4 demonstrating a good understanding of Verilog syntax and structure. However, even the most advanced LLMs struggle when dealing with the long-range dependencies and complex interactions present in large chip design projects. Open-source models, while showing potential, still lag behind. RTL-Repo offers a crucial tool for researchers to evaluate and improve LLMs for hardware design. It also highlights the challenges that remain in applying AI to this complex domain. As LLMs continue to evolve, benchmarks like RTL-Repo will be essential in guiding their development and unlocking their full potential for chip design automation.