Large language models (LLMs) are increasingly used in scientific research, offering potential breakthroughs in fields like chemistry. However, their tendency to generate inaccurate or unsafe responses, especially regarding hazardous materials, raises serious concerns. A new benchmark called ChemSafetyBench aims to address this issue by evaluating the safety and accuracy of LLMs in the chemistry domain. ChemSafetyBench focuses on three key tasks: querying chemical properties, assessing the legality of chemical uses, and describing synthesis methods. Each task requires progressively deeper chemical knowledge and involves over 30,000 samples across diverse chemical materials. Researchers incorporated handcrafted templates and 'jailbreaking' scenarios—attempts to trick the LLM into providing unsafe information—to ensure a robust evaluation. The benchmark uses an automated framework and even employs another LLM (GPT) as a judge to evaluate responses for correctness, safety, and appropriateness. Initial tests with state-of-the-art LLMs, including GPT-4 and several open-source models, revealed significant vulnerabilities. The models struggled to accurately assess chemical safety, often providing incorrect or misleading information. Interestingly, the study also found that some models' seemingly high performance stemmed from biased random guessing rather than true chemical understanding. This research highlights two key challenges: the fragmented nature of chemical terms when processed by LLMs and the lack of specialized chemical knowledge in their training data. Researchers found that LLMs often break down complex chemical names into small, meaningless fragments. Further, standard chemical information is often locked behind paywalls and not readily accessible in the public data used to train these models. To explore solutions, the researchers experimented with enhancing LLMs using external tools like Google Search and Wikipedia. Early results suggest that access to external knowledge can improve LLM performance in chemistry. This points towards future developments focusing on specialized training datasets and the integration of reliable external knowledge sources. Ensuring LLM safety in chemistry, and other scientific domains, will require ongoing collaboration between AI experts and domain specialists. Building robust safeguards against malicious prompting and improving evaluation methods are crucial steps towards realizing the full potential of AI in scientific discovery while mitigating potential risks.