Imagine effortlessly controlling your smart home, ordering food, or booking a taxi with simple voice commands. This seamless interaction with technology is the promise of task-oriented queries, where AI agents understand and execute user requests. However, accurately evaluating and optimizing the quality of these AI agents requires a reliable benchmark—a standardized test to measure how well they understand and fulfill user requests. Introducing ToQB, a novel approach to benchmark generation for task-oriented queries. Researchers have devised an automated method that transforms complex, multi-turn dialogues into concise, one-shot queries. Think of it as summarizing a lengthy conversation into a single, actionable command. This method leverages existing dialogue datasets and employs a large language model (LLM) to extract and refine user intents. ToQB offers a powerful tool for assessing the performance of AI agents in diverse scenarios. By providing a standardized measure, ToQB can help researchers and developers identify areas for improvement and push the boundaries of AI capabilities. Beyond voice assistants, ToQB can be applied to search engines, chatbots, and other LLM-based services, enabling them to better understand and execute complex requests. It’s about making AI truly helpful in our everyday lives, from simple tasks to complex interactions. The ToQB project is open for contributions, inviting the research community to expand the benchmark to new domains and languages. This collaborative effort will accelerate the development of reliable and user-friendly AI, unlocking the potential of task-oriented queries in a world increasingly reliant on intelligent technologies.