Imagine an AI that could read scientific papers, understand complex biological processes, and even fact-check existing knowledge bases. That's the ambitious goal of researchers developing "AI scientists," autonomous agents designed to accelerate biomedical discovery. But how do you evaluate such a complex system? A new benchmark called BioKGBench aims to answer just that. Traditional methods often fall short, relying on simple question-answering that doesn't reflect a scientist's true abilities. BioKGBench takes a different approach, testing two core skills: understanding scientific literature and interacting with structured knowledge graphs. It's like giving an AI a pop quiz on both textbook knowledge and the ability to interpret research findings. The benchmark introduces a novel task called "KGCheck," where the AI agent has to identify errors in existing biomedical knowledge graphs. This involves querying the knowledge graph, retrieving relevant information from research papers or databases, and then determining whether the information in the graph is supported or refuted by external evidence. It's a challenging task that mirrors the process of scientific review, where researchers constantly scrutinize and validate each other's findings. Researchers tested various large language models (LLMs), both open-source and commercial, on BioKGBench. Surprisingly, even state-of-the-art AI agents struggled with these tasks, highlighting the need for better tools and training methods. To address this, the researchers developed a baseline agent called BKGAgent. This agent uses a multi-agent framework, simulating a research team with a leader and specialized assistants. One agent queries the knowledge graph, another verifies the information, and the leader oversees the process and makes the final decision. While BKGAgent is a promising start, the results revealed some interesting challenges. The leader's reasoning abilities played a crucial role, and errors in its judgment often led the team astray. Additionally, while the agents were good at selecting the right tools for the job, they sometimes struggled to synthesize the information effectively, highlighting the complexities of scientific reasoning. BioKGBench represents a significant step toward evaluating and improving AI agents for biomedical science. It provides a challenging and realistic testbed, pushing the boundaries of what AI can achieve in this critical field. As AI scientists become more sophisticated, they could revolutionize how we conduct research, accelerating the pace of discovery and ultimately leading to breakthroughs in healthcare and disease treatment.