Grading student essays is a time-consuming task for teachers. Could artificial intelligence help lighten the load? A new study explores whether large language models (LLMs) can accurately assess essays, comparing their performance to human teachers. Researchers analyzed how well different LLMs, including GPT-3.5, GPT-4, and the new o1 model, graded German student essays based on ten criteria, ranging from plot logic and expression to spelling and punctuation. They found that closed-source models like GPT-4 and o1 were more reliable than open-source alternatives, showing stronger agreement with teacher evaluations, especially when it came to language-focused aspects. Interestingly, the o1 model achieved a remarkable correlation of 0.74 with human assessments for overall essay scores. However, the study also revealed some key limitations. LLMs tend to give more lenient grades than human teachers, particularly struggling to evaluate content-related aspects like plot logic and the structure of the main body. This suggests that while LLMs show promise for automating parts of essay grading, especially the tedious parts related to language mechanics, they still lack the nuanced understanding needed to fully replace human judgment when it comes to evaluating the quality and depth of an essay’s content. As LLM technology advances, future research might explore how these models can be refined to better capture the complexities of content evaluation and become even more useful tools for educators.