Imagine an AI meticulously reviewing a radiologist's report, line by line, offering corrections and even grading the severity of any errors. That's the essence of FineRadScore, a new tool designed to assess the quality of AI-generated radiology reports. Why is this important? AI is increasingly used to assist radiologists, but ensuring the accuracy of these AI-generated reports is crucial. Traditional methods of evaluation are time-consuming and expensive, often requiring expert radiologists to manually review each report. FineRadScore automates this process, using a large language model (LLM) to compare an AI-generated report against a 'ground truth' report. It not only identifies discrepancies but also suggests corrections, assigns severity scores to errors (ranging from 'not actionable' to 'emergent'), and even provides explanations for its assessments. Researchers tested FineRadScore on various datasets, including reports generated by different AI models and covering different sections of radiology reports (findings and impressions). The results are promising: FineRadScore accurately identifies the type of correction needed (deletion, rewriting, insertion) and generates text that closely matches expert corrections. It also aligns well with radiologist judgments on error severity. While FineRadScore excels in many areas, it faces challenges when reports are stylistically different, sometimes over-correcting based on phrasing rather than clinical meaning. Future research aims to refine its ability to discern clinically relevant errors from stylistic variations. FineRadScore represents a significant step towards automating the evaluation of AI-generated radiology reports, offering a detailed, efficient, and scalable solution. This technology has the potential to improve the reliability of AI in radiology, ultimately leading to more accurate diagnoses and better patient care.