Large language models (LLMs) are making waves in software development, but how do we know if they’re truly up to the task? A new research paper from Polytechnique Montréal introduces HardEval, a framework for rigorously testing the difficulty of coding challenges for LLMs. Current benchmarks often give a broad overview, but they don’t tell us how challenging individual tasks are. A 90% score on a benchmark full of easy tasks is less impressive than a 90% score on truly difficult problems. HardEval dives deeper, using a variety of prompts and multiple LLMs to pinpoint the truly hard problems. For each coding task, HardEval generates prompts with varying levels of detail and different phrasing. It then uses several LLMs to try and solve the problem, measuring not just if the code works, but also how similar it is to correct solutions. This gives a much more nuanced 'difficulty score' than simple pass/fail metrics. The researchers tested this on HumanEval+ and ClassEval, two popular code generation benchmarks. They found that a surprisingly small percentage of tasks—just 21% and 27% respectively—are actually hard for LLMs. They also found intriguing differences in how various LLMs handle certain types of problems, suggesting that some models may be better suited to particular coding tasks. Perhaps most interestingly, HardEval doesn’t stop at assessment. It can also help create *new*, targeted challenges based on the types of problems that consistently stump LLMs. This means we can generate benchmarks focused on specific areas where LLMs need to improve. HardEval offers a valuable tool for researchers and developers alike. It’s a way to not just evaluate how good LLMs are at coding, but also to understand their weaknesses and create increasingly challenging tests to push the boundaries of AI-powered software development.