The rapid evolution of AI demands equally swift progress in how we evaluate these powerful models. Existing benchmarks often become outdated or too narrow to capture the full spectrum of an AI's capabilities, especially with the rise of generative models. This is where BENCHAGENTS comes in, offering a groundbreaking approach to automated benchmark creation. Imagine a team of specialized AI agents working together, planning, generating, verifying, and evaluating test data, all while learning from human feedback. That's the core idea behind BENCHAGENTS, a multi-agent framework that leverages large language models (LLMs) to build dynamic, high-quality benchmarks for complex AI tasks. Traditional benchmark creation is slow, expensive, and relies heavily on human annotation. BENCHAGENTS automates this process by dividing it into four key stages, each handled by a dedicated LLM agent. A Planning Agent designs the overall benchmark strategy, defining parameters and constraints. A Data Generation Agent then brings this plan to life, creating diverse test instances. A Verification Agent acts as quality control, ensuring the generated data meets specific criteria. Finally, an Evaluation Agent designs the metrics and methods for assessing model performance. This collaborative agent approach offers significant advantages. It allows for fine-grained control over data diversity and quality, ensuring the benchmark truly tests the target capabilities. The framework also incorporates human-in-the-loop feedback, allowing developers to guide the agents and refine the benchmark at each stage. This blend of automated efficiency and human oversight results in benchmarks that are both comprehensive and relevant. To demonstrate its power, BENCHAGENTS was used to create two benchmarks focusing on complex generative tasks: calendar scheduling and constrained long-form text generation. These benchmarks were then used to evaluate seven state-of-the-art LLMs, revealing intriguing insights into their strengths and weaknesses. For example, the evaluations showed that while many LLMs can handle individual constraints, they often struggle to satisfy multiple constraints simultaneously. The benchmarks also highlighted the difficulty LLMs face with numerical and logical reasoning, especially when tracking state across multiple steps. BENCHAGENTS represents a significant step towards more robust and scalable AI evaluation. By automating the benchmark creation process while retaining human oversight, it empowers researchers to keep pace with the rapid advancements in AI. This opens doors to more comprehensive evaluations, leading to a deeper understanding of AI capabilities and paving the way for more responsible and effective AI development. While challenges remain, such as computational costs and the potential for LLM biases, BENCHAGENTS offers a promising framework for building the next generation of AI benchmarks.