Imagine having an AI assistant that could design revolutionary new materials, atom by atom. This dream is driving researchers to explore how large language models (LLMs), renowned for their text-processing abilities, can be applied to the complex world of materials science. However, new research reveals a significant hurdle: LLMs struggle to grasp the 3D geometry crucial for understanding material properties. The study, introducing a new benchmark called "MatText," reveals that simply feeding LLMs more text describing materials isn't enough. Instead, these models tend to over-rely on simpler information like chemical composition, even when given explicit geometric data. Think of it like trying to understand a building's stability based only on the types of bricks used, ignoring the blueprint entirely. This surprising finding challenges the notion that bigger models and datasets automatically lead to better performance in scientific domains. The MatText benchmark systematically evaluates nine different ways to represent materials as text, including some designed to emphasize key physical insights like bonding and local atomic environments. However, even with these enhanced representations, current LLMs fall short. They excel at learning local arrangements of atoms, but fail to capture the overall structure, highlighting the need for innovative approaches to geometric reasoning in AI. This limitation isn't just an academic curiosity; it directly impacts the potential for AI-driven materials discovery. While LLMs might excel at generating hypothetical materials, their ability to accurately predict their properties is currently hampered. Future research will focus on developing new ways to represent geometric information that are more compatible with how LLMs learn, paving the way for AI that truly understands the intricate dance of atoms and molecules.