Imagine an AI that not only solves complex math problems but also *invents* new ones that stump even the smartest large language models (LLMs). That's the intriguing idea explored in "AI-Assisted Generation of Difficult Math Questions." Researchers have designed an innovative framework that combines the strengths of LLMs and human ingenuity to create a dataset of exceptionally challenging math problems, dubbed MATH². Why is this important? Because current methods of evaluating LLMs on math are hitting a wall. Existing datasets are getting stale—LLMs have essentially memorized them—and relying on human experts to craft new, difficult problems isn't scalable. This new research addresses the problem head-on. The process starts by using LLMs to extract core "skills" from existing math problems. These skills, like "ratio and proportion" or "geometric series," become the building blocks for novel questions. The trick? The system pairs up seemingly unrelated skills—say, combining area calculations with prime number knowledge—forcing the LLM to think "outside the box." This clever technique results in problems that are not only difficult but also diverse and engaging for human learners too. The AI doesn't work in isolation. Human experts play a key role in refining the AI-generated questions, making them even more intricate and insightful. This collaboration results in high-quality questions that often stump the very LLMs that helped create them. Interestingly, the research uncovered a curious pattern: an LLM's success rate on MATH² is roughly the square of its success rate on the original MATH dataset. This suggests that MATH² problems genuinely require the mastery of *two* distinct mathematical skills, making them a more robust test of AI reasoning. While the initial results are promising, the current process relies heavily on powerful (and expensive) LLMs like GPT-4. Future work aims to make the process more efficient by incorporating open-source models and automated validation techniques. The ultimate goal is a system that can automatically generate diverse and challenging math problems across a wide range of difficulty levels, keeping LLMs—and their human evaluators—on their toes. This could also have broader implications for education, providing a new way to design challenging and engaging math curricula for students at all levels. Imagine an AI tutor that customizes problems based on your individual strengths and weaknesses – the possibilities are exciting.