Large language models (LLMs) have become remarkably adept at following instructions, opening up exciting new possibilities for human-computer interaction. But what about the systems designed to *find* information? Are retrieval models keeping pace with these advancements? New research suggests they might be lagging behind. A recent study delves into this very question, exploring whether retrieval models truly grasp and respond to user instructions beyond simply matching keywords. Researchers have developed a new benchmark called InfoSearch, specifically designed to test how well retrieval models adhere to instructions related to six key document attributes: target audience, keywords, format, language, length, and source. The benchmark also cleverly incorporates “reverse instructions” to ensure models aren't just picking up on superficial cues. To get a more precise measure of instruction following, the study introduces two innovative metrics: Strict Instruction Compliance Ratio (SICR) and Weighted Instruction Sensitivity Evaluation (WISE). These metrics provide a granular look at how well models adhere to instructions, particularly in the crucial top search results. The findings reveal a mixed bag. While reranking models generally outperform traditional retrieval models in following instructions, even the most advanced models struggle with certain attributes like document format and target audience. This suggests a need for more sophisticated training methods that go beyond simple keyword matching and incorporate a deeper understanding of context and user intent. The ability to fine-tune models with instruction-specific data shows promise, as does simply increasing model size. However, the research makes it clear that there's still significant room for improvement before retrieval models can truly claim to be instruction-aware. The research underscores the growing importance of aligning information retrieval with the rapid progress in LLMs. As users become accustomed to interacting with AI through natural language instructions, retrieval systems must evolve to meet these expectations. This study serves as a valuable benchmark and a call to action for researchers and developers to focus on building truly instruction-following retrieval models that can deliver the precise information users need, in the way they want it.