Large Language Models (LLMs) are increasingly used for various tasks, including judging the grammatical correctness of text. But how do we accurately assess their abilities? A new research paper explores effective ways to tap into the grammatical knowledge of LLMs to enhance acceptability judgments, going beyond simply comparing sentence probabilities. Researchers explored nine different methods across six LLMs in both English and Chinese. They found that embedding the sentences in context within a template before inputting to the LLM (in-template LP) and a prompting-based method (Yes/No probability computing) achieved the best performance, sometimes even outperforming humans! Interestingly, the two methods excel in different linguistic areas, suggesting they access distinct facets of LLMs' grammatical knowledge. For example, the Yes/No method shows resilience against token-length bias, meaning its performance doesn’t falter when comparing sentences of significantly different lengths. While the other methods often struggle when the correct sentence is longer. While these advancements are impressive, some challenges remain, including accurately judging sentences where word order is the sole issue. This suggests LLMs don’t fully grasp how word shuffling impacts grammar. The key takeaway? Ditch the conventional probability comparison methods. Embrace a multi-pronged approach using both in-template and prompting methods to unlock the true potential of LLMs as grammar gurus.