Imagine a world where AI could assess the novelty of research papers, filtering the groundbreaking from the incremental. This is the challenge posed by SchNovel, a new benchmark designed to test the ability of Large Language Models (LLMs) to judge the novelty of scholarly publications. Traditional methods of evaluating LLM creativity often focus on semantic novelty—how unique or unusual the generated text is. But scholarly novelty is different. It's about pushing the boundaries of knowledge, introducing new ideas, methods, or insights that build upon existing research. SchNovel tackles this by presenting LLMs with pairs of research papers from six different fields, ranging from computer science to quantitative finance. The LLM must determine which paper is more novel, using only the title, abstract, and metadata—mirroring the information available to a human reviewer. The results are fascinating. While some LLMs, like GPT-4, demonstrate promising abilities, the research reveals that these models still struggle with the nuances of scholarly novelty, especially in fields like mathematics and physics. One key innovation tested was RAG-Novelty, a method that simulates the human review process by retrieving similar papers to provide context for novelty assessment. This approach showed significant improvements, suggesting that giving LLMs access to relevant prior work enhances their judgment. However, the research also uncovered biases. LLMs exhibited a preference for papers from prestigious universities, highlighting the potential for such systems to perpetuate existing inequalities in research if not carefully designed. This exploration into AI-driven novelty assessment is just the beginning. Future research aims to expand the benchmark to include more fields and papers, testing which parts of a paper best convey its novelty to LLMs. Ultimately, this work has important implications for the future of peer review, the dissemination of scientific knowledge, and the role AI will play in shaping the landscape of scholarly research. Can AI truly judge the novelty of research? SchNovel provides a crucial first step in answering this complex question, revealing both the potential and the pitfalls of applying AI to this critical task.