Imagine an AI that could not only understand complex chemical research papers but also design new molecules and predict reaction outcomes. While this might sound like science fiction, large language models (LLMs) are steadily making their way into chemistry research. But how good are they really? A new benchmark called ChemEval is putting these LLMs under the microscope, testing their chemical knowledge across a wide range of challenges, from basic concepts to advanced scientific deduction. Researchers from the University of Science and Technology of China have created a rigorous, multi-level evaluation to examine what LLMs can actually do in chemistry. ChemEval throws 42 distinct tasks at the models, categorized into four key areas: understanding basic chemical knowledge, deciphering research papers, understanding molecules, and performing scientific reasoning. This comprehensive benchmark uses data carefully crafted by chemical experts, ensuring a real-world focus for evaluating LLMs. In testing popular LLMs like GPT-4, Claude, and specialized chemistry models, the research revealed a mixed bag. While general-purpose models like GPT-4 shone at understanding research papers and following instructions, they stumbled when faced with questions requiring deeper chemical knowledge. Specialized models, on the other hand, showed stronger chemistry skills but lagged behind in general language tasks. This suggests a trade-off: breadth of knowledge versus specialized expertise. One intriguing finding is the challenge LLMs face with molecular name translation. Converting between different ways of representing molecules, such as IUPAC names and SMILES strings, proved tricky for most models. This is because LLMs are trained on text and struggle with the strict formats needed for chemical formulas. The study also looked at how few-shot learning—giving models a small number of examples before asking a question—affected performance. While it boosted language comprehension for some models, it had little effect on complex reasoning tasks. This points to the difficulty LLMs have in grasping the deeper, expert-level reasoning of chemistry. The researchers also found that, in general, larger models performed better, but even they have a long way to go before they can genuinely assist chemists in complex research. ChemEval provides a crucial stepping stone for improving LLMs in chemistry. It highlights where current models fall short and sets the stage for developing more specialized and powerful AI tools for chemical research. This work opens exciting possibilities for the future. Imagine LLMs that can accurately predict reaction outcomes, design optimal synthesis pathways, or even discover new materials. While not there yet, ChemEval marks an important advance toward realizing AI's potential in revolutionizing chemical discovery.