Imagine training an AI to understand the complex language of astronomy research. Could it identify key concepts like "black holes" or "dark matter" as accurately as a human expert? That's the challenge tackled in a new research paper exploring whether Large Language Models (LLMs), specifically GPT-3, can be effective domain expert annotators for astronomy literature. Researchers investigated how well GPT-3, in both its original and fine-tuned forms, could identify scientific entities in astronomy article titles. They compared its performance to human annotators, both experts and non-experts, using a specialized annotation scheme focusing on research contributions. The results reveal a fascinating dynamic. While fine-tuning significantly improved GPT-3's ability to identify entities, it still fell short of human expert performance. Interestingly, non-experts assisted by GPT-3 achieved moderate agreement with the expert, suggesting that AI can be a valuable tool, even if it can't fully replace human expertise. The study also highlights the inherent subjectivity of scientific entity annotation, particularly in specialized fields like astronomy. Even among human experts, there's room for interpretation. This research underscores the ongoing challenge of creating high-quality labeled datasets for training AI models in specialized domains. While AI tools like GPT-3 offer promising assistance, the need for human expertise, especially in complex fields, remains crucial. The publicly available dataset generated from this research will be a valuable resource for future NLP research in astronomy, paving the way for more sophisticated AI tools that can unlock the secrets of the universe hidden within scientific literature.