Imagine an AI assistant that not only understands your current progress in, say, assembling a bookshelf, but also suggests your next steps. Researchers at Meta AI and UNC Chapel Hill are bringing this closer to reality with VidAssist, an AI system that generates step-by-step plans from instructional videos. Their research tackles the challenge of anticipating sequences of actions towards a goal—like a recipe or DIY project—directly from visual demonstrations. Traditional approaches struggle to generalize to new tasks, but VidAssist leverages large language models (LLMs) for a more flexible approach. It works like this: the system first translates video or images of a task into text descriptions. Then, it enters a 'propose, assess, search' loop. The LLM proposes possible next steps, a set of specialized functions assess how well these align with the ultimate objective, and a search algorithm dynamically explores the most promising action sequences. Essentially, the LLM acts as both a knowledge base and a judge of its own suggestions. This approach avoids the trap of overfitting to limited training data and can even perform well in situations with little or no specific examples. VidAssist achieved state-of-the-art results, outperforming fully supervised models in predicting action sequences. This research has far-reaching implications for AI assistance in everyday tasks. Imagine an app that provides dynamic guidance during cooking or guides a robot to build furniture. While challenges like enhancing long-term planning and improving visual recognition remain, VidAssist represents an exciting step toward truly intelligent assistants that can understand and guide our actions in the real world.