Imagine a world where mangoes are blue. Sounds like a sci-fi movie, right? But this whimsical scenario is the basis for NEMO, a new benchmark designed to test the limits of AI’s visual reasoning. Researchers from the University of Tokyo explored how well Multimodal Large Language Models (MLLMs), AI systems that combine language and image understanding, can identify objects when their attributes are drastically changed. They found that even the most advanced MLLMs struggle when presented with these “beyond-commonsense” objects. For example, while many MLLMs can correctly identify a regular mango, they falter when shown a picture of a blue one. This reveals a significant gap in AI's ability to reason like humans. We adapt to such changes effortlessly because we possess prior knowledge and adaptable perception. We know a mango is still a mango, even if it's blue. But for AI, this simple task proves challenging. The research suggests that stronger vision encoders, the components of MLLMs that process images, generally improve performance. However, simply making the AI models bigger doesn't always lead to better results. In fact, the researchers found that scaling up the model size can sometimes weaken the vision encoder during the fine-tuning process, effectively making the AI less perceptive. This research is crucial for building more robust and reliable AI. Imagine self-driving cars that can navigate unexpected situations, like a road covered in bright pink snow, or medical imaging AI that can still identify anomalies even if the image quality is poor. NEMO highlights the current limitations of MLLMs and provides valuable insights for developing future AI systems that can reason and adapt to changes in the real world, just like we do.