Large language models (LLMs) are rapidly becoming integrated into various applications. However, their increasing use also exposes these systems to security risks, such as data breaches and manipulation through malicious prompts. Open-source LLM vulnerability scanners have emerged as a vital tool for automated red-teaming, helping identify and mitigate these risks. This post delves into a comparative analysis of four leading open-source scanners: Garak, Giskard, PyRIT, and CyberSecEval, exploring their strengths, weaknesses, and a critical reliability gap that needs urgent attention. These scanners employ a common architecture, generating adversarial prompts to elicit potentially harmful responses from the target LLM and then evaluating the success of these attacks. They differ, however, in their approach to generating and evaluating attacks. Garak boasts the broadest coverage with a research-backed, static attack dataset, while Giskard offers flexible, customizable attacks using both static and LLM-based methods, including a unique dual-context mechanism for tailoring tests. PyRIT provides a fully LLM-based framework with multi-turn attack capabilities, allowing for dynamic interactions with the target model. CyberSecEval, on the other hand, specializes in detecting vulnerabilities in LLM-generated code, focusing on insecure coding practices and malicious code generation. Our quantitative analysis reveals a significant reliability issue across all four scanners. While Garak's attacks demonstrated the highest effectiveness, its evaluator showed a concerning margin of error. Similarly, despite PyRIT and Giskard’s more reliable LLM-based evaluators, they sometimes misinterpret instructions or generate unexpected requirements, leading to incorrect classifications. This highlights a critical challenge: the evaluators themselves are prone to errors, impacting the accuracy of vulnerability assessments. Qualitative examples reveal that static evaluators often lack contextual awareness, while LLM-based evaluators can misinterpret their instructions or exhibit uncontrolled reasoning. The lack of transparency in LLM-based evaluations further complicates the process of understanding and addressing these errors. This research underscores the need for improved quality standards and a unified benchmarking framework for LLM vulnerability scanners. Standardized evaluations of the evaluator component are crucial to ensure accurate vulnerability detection. A common platform for benchmarking scanners would foster transparency, facilitate targeted improvements, and ultimately contribute to building more robust and secure LLM systems. As LLMs become increasingly central to our digital world, enhancing the reliability of these security tools is paramount for mitigating risks and ensuring responsible AI development.