Imagine an AI assistant that could instantly sift through hundreds of thousands of astronomy papers to answer your burning questions about the universe. That's the tantalizing possibility explored by researchers in a new study designing a framework for evaluating Large Language Models (LLMs) in astronomy. Traditionally, astronomy research involves painstakingly combing through vast databases like arXiv and ADS to find relevant publications. This new research proposes a dynamic evaluation framework centered around a Slack chatbot powered by Retrieval-Augmented Generation (RAG). This chatbot can answer complex astronomy questions by retrieving and processing information from a massive dataset of arXiv papers. Users interact with the bot directly within Slack, asking questions and providing feedback through upvotes, downvotes, and comments. This real-world interaction data is then collected and anonymized, offering valuable insights into how astronomers use and perceive the helpfulness of LLMs. The researchers envision this framework as a way to dynamically assess the strengths and weaknesses of LLMs in a real-world research setting. By analyzing user interactions, they hope to understand how different research topics influence the types of questions asked, how LLM performance varies across subfields, and whether astronomers find these tools genuinely useful. This research is just the beginning. The team plans to release the collected data and evaluation results in a future paper. This innovative approach promises to not only streamline astronomy research but also to refine and improve the capabilities of LLMs for scientific discovery. The future of astronomy research might be just a chat away.