The rapid evolution of Large Language Models (LLMs) demands a continuous evolution of evaluation methods. Existing benchmarks, often relying on static datasets, struggle to keep pace with these advancements. This is where IDGen, an Item Discrimination-induced prompt generation framework, comes in. IDGen brings dynamism to LLM evaluation by creating test data that adapts to increasing LLM capabilities. Inspired by Item Discrimination (ID) theory from educational assessment, this framework ensures that each question effectively differentiates between higher and lower-performing LLMs. IDGen prioritizes both breadth and specificity in its data generation, comprehensively evaluating LLMs while highlighting their relative strengths and weaknesses across diverse tasks and domains. This is achieved through a clever combination of “instruction gradient” and “response gradient” methods. The former leverages predefined rules to guide the generation of questions, ensuring adherence to specific content and maintaining diversity. The latter refines questions based on LLM responses, leading to more complex and nuanced challenges. A key innovation of IDGen is its self-correction mechanism. For general text questions, this involves assessing safety, neutrality, integrity, and feasibility. Mathematical questions undergo a more rigorous 'Chain of Thought' (CoT) check to ensure logical soundness and precision, avoiding common pitfalls like contradictions and unsolvable problems. To set a reliable benchmark, IDGen employs a multi-model voting system for reference answers. General text responses are evaluated based on safety, correctness, relevance, comprehensiveness, readability, richness, and humanization. For mathematical questions, responses from multiple LLMs are compared, with expert mathematicians providing further validation. IDGen introduces two key metrics: prompt discrimination power and prompt difficulty. These metrics, along with other prompt features like category and length, are used to train models to predict discrimination and difficulty levels. This allows for an automated and standardized way to assess the quality of generated questions. In comparison with existing datasets like SELF-INSTRUCT and WizardLM, data generated by IDGen proved more challenging and discriminative, resulting in a lower average score and higher variance across evaluated LLMs. This signifies IDGen’s effectiveness in distinguishing between models of varying capabilities. IDGen, with its adaptive, self-correcting, and multi-faceted approach, presents a significant step forward in LLM evaluation. Its publicly available dataset of over 3,000 prompts, along with the discrimination and difficulty estimation models, provides valuable tools for the community. While reliance on the performance of existing LLMs and the need for improved accuracy in complex mathematical reasoning present ongoing challenges, IDGen establishes a new benchmark for evaluating the ever-evolving capabilities of LLMs.