Imagine harnessing the power of AI to sift through the deluge of tweets during a crisis, pinpointing vital information amidst the noise. That's the ambitious goal researchers tackled in a recent study using large language models (LLMs) like GPT-4, Gemini, and Claude. They wondered: could these AI powerhouses analyze tweets and determine, with zero prior training, which ones contained crucial details about a disaster? The challenge? Teaching AI to understand nuances of language, filter out irrelevant chatter, and identify truly informative tweets in the chaos of a crisis. The team used a benchmark dataset called CrisisBench, a collection of tweets from various disaster events. They tested how well the LLMs could identify "informative" tweets—those actually helpful in a disaster—and classify them into 16 categories, from "infrastructure damage" to "missing persons." Surprisingly, the AI performed admirably on the basic task of identifying informative tweets, rivaling models specifically trained for the task. But when it came to the more complex classification, things got trickier. Why? Turns out, the definitions of categories within the dataset sometimes clashed. What one dataset called "affected individuals" another might classify as a "personal update." These inconsistencies tripped up the LLMs, which relied on clear definitions for accurate classification. The research highlights a critical challenge in using LLMs for crisis response: the need for clear, consistent data. While the AI showed impressive potential, its success is tied to the quality of the information it receives. This study serves as a stepping stone for refining how we use AI during emergencies, paving the way for more accurate, real-time information processing that could ultimately save lives.