Imagine an AI assistant that not only understands your questions but also *sees* your progress and offers guidance, much like a helpful friend in real life. That’s the groundbreaking concept behind the “Show and Guide” research, which introduces a new multimodal AI model called MM-PlanLLM. Traditional AI excels at text, but real-world tasks like cooking or assembling furniture are inherently visual. This new model bridges the gap, using both text instructions (like a recipe) and visual cues (like a photo of your current progress) to provide dynamic, context-aware assistance. For instance, if you’re following a recipe and unsure what to do next, you could simply snap a picture of your dish. MM-PlanLLM would analyze the image, determine which step you're on, and provide the next instruction, even if you're not following the recipe precisely. This is possible thanks to MM-PlanLLM’s unique ability to learn cross-modal representations—linking text and visual information within the context of a task. This model is trained in stages, starting with basic image-caption matching and then progressively learning more complex tasks like video moment retrieval (showing you the relevant part of an instructional video) and visually-informed step generation (telling you what to do next based on an image). While current AI assistants mostly interact through text, this research paves the way for more intuitive, truly helpful AI companions that understand both what you say and what you see. However, challenges remain. MM-PlanLLM’s current context window is limited, and broader multimodal support, like answering complex visual questions, is still in its early stages. Nevertheless, this research offers an exciting glimpse into the future of AI—one where your AI assistant not only answers but also sees, guides, and collaborates with you in the real world.