Imagine an AI agent that could analyze medical images, interpret complex reports, and even suggest treatment plans, much like a human radiologist. This intriguing possibility is precisely what researchers explored in a new study introducing "RadABench," a comprehensive benchmark designed to test the capabilities of large language models (LLMs) as the "brains" of such AI agents in radiology. The core question? Can today's advanced LLMs effectively navigate the intricate world of radiology, understanding tool descriptions, translating clinical queries into actionable steps, and orchestrating the use of various tools to perform complex analyses? RadABench creates a simulated radiology environment, complete with synthetic patient records, diverse imaging modalities, a range of analytical tools, and a variety of clinical tasks. Seven leading LLMs, including both closed-source models like GPT-4 and open-source alternatives like LLaMA, were put to the test. The results reveal a mixed bag. While these LLMs showed promise in certain straightforward tasks, they stumbled when faced with more complex scenarios. They struggled with tasks requiring multiple steps, often failing to select the optimal tools or to correctly interpret and integrate information from various sources. A key weakness identified was the LLMs' difficulty in understanding the nuanced descriptions of specialized radiology tools and in managing the flow of information between different stages of analysis. Interestingly, closed-source models generally outperformed their open-source counterparts, suggesting that the vast resources and proprietary training data of companies like Google and OpenAI still give them an edge. This research highlights both the exciting potential and the significant challenges that lie ahead in developing truly capable AI agents for radiology. While a fully autonomous AI radiologist is still a distant prospect, the study provides valuable insights into the steps needed to bridge the gap between current LLM capabilities and the complex demands of real-world clinical practice. The open-source nature of RadABench also offers a crucial platform for researchers to continue refining and improving these AI agents, bringing us closer to a future where AI can play a more significant role in healthcare.