Imagine an AI not writing code, but judging it. That's the intriguing premise behind CodeJudge-Eval, a new benchmark designed to test whether Large Language Models (LLMs) truly understand code, or just mimic it. Instead of generating code from prompts, LLMs are tasked with evaluating existing code solutions for correctness, identifying errors like wrong answers, time-outs, or compilation issues. This approach challenges LLMs to go beyond memorizing patterns and demonstrate deeper code comprehension. Researchers tested 12 leading LLMs, including both proprietary and open-source models, on CodeJudge-Eval. The results? Even the most advanced models struggled, exposing a gap between their code generation skills and their capacity for critical evaluation. While proprietary models generally fared better, most open-source LLMs underperformed compared to random guessing. Surprisingly, an LLM's ability to generate the correct code didn't guarantee its ability to accurately judge another solution to the same problem. This suggests that code generation and code judging tap into different skill sets. This research raises important questions about how we evaluate LLM coding abilities. CodeJudge-Eval offers a fresh perspective, pushing beyond traditional benchmarks and revealing new insights into the limitations and potential of LLMs in code understanding. While the benchmark currently focuses on specific coding tasks, it paves the way for exploring AI's capacity for logical analysis and problem-solving across other domains.