Unit testing, the often tedious yet essential practice of writing small tests to ensure code behaves as expected, is ripe for automation. Could large language models (LLMs) finally be the answer? A new benchmark, TestGenEval, aims to find out. TestGenEval throws a real-world challenge at LLMs. Instead of simple, self-contained code snippets, it uses 68,647 real tests from 1,210 code files across 11 popular Python projects. The benchmark examines two key tasks: generating a full test suite from scratch, mirroring a developer starting from a blank slate, and completing existing test suites, simulating a developer adding tests to improve coverage. The results? LLMs still have a long way to go. Even the most powerful model, GPT-4o, only achieved 35.2% average code coverage in the full test generation task. This means the AI-generated tests only exercised about a third of the code's possible execution paths. The models fared better at test completion, but still struggled to add meaningful tests that boosted code coverage when existing tests were already comprehensive. Why are LLMs falling short? The analysis points to a few key weaknesses. They often struggle to reason about how the code actually executes, leading to incorrect assertions—claims about what the code *should* do that don't match reality. They sometimes miss subtle dependencies or interactions within the code, resulting in tests that time out or generate errors. And, surprisingly, they sometimes fail to include assertions altogether, producing tests that don't actually test anything. While TestGenEval highlights current limitations, it also provides a crucial stepping stone for future research. By simulating real-world testing scenarios, this benchmark allows researchers to pinpoint areas for improvement and develop more robust, practical AI-powered testing tools. The next generation of coding assistants could very well learn from this challenge and finally take the burden of unit testing off developers' shoulders.