Imagine a world where designing new materials—for anything from faster computer chips to more efficient solar panels—is as easy as typing a description into a computer. That's the tantalizing promise of using large language models (LLMs) in materials science. LLMs, the technology behind AI chatbots like ChatGPT, are already shaking up fields like medicine and coding. But can they really predict the properties of materials accurately? A new, massive benchmark called LLM4Mat-Bench aims to find out. This benchmark is the biggest of its kind, containing nearly 2 million crystal structures and encompassing a vast range of material properties. Researchers put various LLMs to the test, including specialized models like LLM-Prop and general-purpose chatbots like Llama 2. Surprisingly, the smaller, specialized LLMs outperformed the giant chatbots by a significant margin. Why? It turns out that general-purpose LLMs sometimes “hallucinate,” generating nonsensical or invalid outputs, especially when confronted with the complex data formats used in materials science. This doesn't mean LLMs are useless for materials discovery. In fact, the research showed that LLMs perform best when given textual descriptions of materials, suggesting that they excel at processing natural language information related to material structures. This opens exciting new avenues for research. The benchmark also revealed that even the most advanced LLMs struggle with certain types of material properties. This highlights the need for further refinement and training of these models. While the dream of on-demand material design isn't here just yet, LLM4Mat-Bench provides a vital step forward in understanding the power and limitations of LLMs in this exciting frontier of science. It paves the way for future research focused on developing more specialized and reliable LLM-based tools for materials property prediction and, ultimately, accelerating the pace of materials discovery.