Imagine teaching a computer to identify key points on any object, like the joints of a human body or the beak of a bird, with just a few examples or even a simple text description. This is the exciting potential of zero- and few-shot keypoint detection, a field pushing the boundaries of computer vision. Traditionally, training AI models for keypoint detection required massive amounts of labeled data, limiting their flexibility and scalability. However, recent advancements leveraging powerful foundation models like CLIP have opened up new possibilities. One of the most promising approaches is OpenKD, a novel model designed for flexible and general keypoint detection. OpenKD can detect keypoints on objects it has never encountered before, using various forms of input: images with keypoint annotations (few-shot learning), textual descriptions (zero-shot learning), or a combination of both. The model achieves this by creating a multimodal prototype set representing visual and textual keypoint features. This allows OpenKD to correlate textual descriptions with visual regions, enabling it to find keypoint locations with remarkable accuracy. Furthermore, OpenKD enhances its ability to understand and detect novel keypoints by incorporating auxiliary keypoints and texts during training. It uses a Large Language Model (LLM) to reason about the relationships between keypoints, greatly improving its spatial reasoning capabilities. This unique approach allows OpenKD to bridge the semantic gap between seen and unseen keypoints, making it more adaptable to new tasks. One of the key innovations of OpenKD lies in its ability to handle diverse and complex textual instructions. Leveraging the parsing capabilities of LLMs, the model can decompose complex text prompts into simpler, actionable instructions. For example, if you ask it to locate the “left eye and nose of a cat,” the LLM will parse this complex instruction into individual components, allowing OpenKD to accurately locate each keypoint. This opens the door for a more natural and flexible interaction with the model. OpenKD’s performance is truly impressive, outperforming existing few-shot and zero-shot keypoint detection methods on several challenging datasets. It demonstrates the power of multimodal learning and showcases the potential of LLMs for reasoning about the visual world. While OpenKD represents a significant step forward, there are still challenges ahead. Further research is needed to improve the quality of auxiliary text reasoning and to explore the interplay between visual and textual prompts. However, OpenKD has opened up exciting new avenues for zero- and few-shot keypoint detection, paving the way for more versatile and powerful AI models in the future.