Automated essay scoring (AES) systems are becoming increasingly popular, but how well do they truly grasp the nuances of good writing? A new research paper challenges the notion that high agreement with human graders equals true understanding. The study dives deep into the decision-making of these AI graders, revealing that while some, like BERT, excel at surface-level features like grammar, they often miss the bigger picture – the logical flow and coherence of an essay. Researchers used a clever technique: they introduced specific changes to essays, like fixing spelling errors or simplifying vocabulary, and observed how the AI graders reacted. The results were surprising. While BERT-like models often overemphasized minor errors, large language models (LLMs) like GPT-4 showed a more nuanced understanding, penalizing disruptions to the essay's organization and logic. This suggests that LLMs, though not perfect, are closer to replicating human reasoning when evaluating essays. The study also found that LLMs can provide insightful feedback that reflects their understanding of these changes, something traditional AES systems can't do. This opens exciting possibilities for using LLMs not just to grade, but to help students improve their writing. However, the research also highlights the need for more sophisticated evaluation methods that go beyond simply measuring agreement with human graders. The future of AES lies in understanding not just *what* these systems score, but *why* they score it that way, paving the way for AI that truly understands the art of writing.