Imagine an AI that could grade medical reports as accurately as a seasoned radiologist. That’s the promise of ER²Score, a groundbreaking new metric designed to assess the quality of automated radiology reports. Generating consistent and accurate radiology reports is a complex challenge for AI. Traditional metrics often fall short, relying on rigid word comparisons that miss crucial nuances in clinical language. ER²Score tackles this problem head-on by using a sophisticated reward model, trained with data generated by the powerful GPT-4 language model. This innovative approach allows ER²Score to understand the subtle differences between high-quality and low-quality reports, mimicking the judgment of human experts. But ER²Score goes further than just assigning a simple pass or fail. It provides detailed sub-scores for various criteria, like accuracy of findings, description of lesions, and even grammar. This granular feedback allows developers to pinpoint areas for improvement in their report generation systems, leading to more accurate and reliable AI-driven diagnostics. The secret sauce behind ER²Score is its unique training process. Using GPT-4, the researchers generated pairs of reports – one “accepted” and one “rejected” – based on their quality. This pairing, combined with a novel “margin-based reward enforcement loss,” trains the AI to distinguish between reports of varying quality, even those with only minor differences. The result is a metric that not only aligns remarkably well with human judgment but is also highly customizable. ER²Score can be adapted to different evaluation criteria, making it a versatile tool for diverse clinical settings. Tests on two datasets showed that ER²Score significantly outperformed traditional metrics in matching expert radiologist evaluations. This advance represents a significant step toward fully automated, high-quality radiology report generation, promising faster, more accurate diagnoses and ultimately, better patient care. While challenges remain, such as further enhancing explainability and scaling up testing, ER²Score paves the way for a future where AI plays a critical role in improving the quality and efficiency of medical reporting.