The rise of multimodal large language models (MLLMs) has opened exciting possibilities for understanding video content in ways never before imagined. But can these AI powerhouses actually *read* videos, deciphering the text within them like humans do? A new benchmark, specifically designed to test video-based optical character recognition (Video OCR) in MLLMs, reveals some surprising truths about the current state of AI. This benchmark presents a novel approach to evaluating how well MLLMs can extract and comprehend text from videos, moving beyond traditional image OCR. It focuses on six key challenges: basic text recognition, understanding the meaning of detected text, spatial relationships between text and objects, recognizing text attributes like font and color, detecting text movement, and pinpointing when text appears and disappears in the video timeline. The researchers constructed this benchmark using a semi-automated process, combining the OCR capabilities of image-based LLMs with human refinement for accuracy and quality. They evaluated several leading MLLMs, including video-specific and image-specific models, to see how they stack up against these challenges. The findings indicate that current video LLMs have difficulty tackling the complexities of video OCR. While some models performed decently in recognizing text, they struggled with more nuanced tasks such as understanding text in motion or locating it within specific timeframes. Surprisingly, even image-based LLMs showed some promise in specific video OCR tasks when evaluated frame by frame, suggesting that their capabilities extend beyond static images. Notably, all models faltered when tasked with pinpointing the exact timing of text appearances, suggesting that temporal awareness remains a significant hurdle for AI. This research highlights the need for continued development and improvement in video LLMs, particularly in handling temporal information and motion tracking. The benchmark provides a crucial tool for researchers, paving the way for more robust and accurate video understanding in the future. As AI continues to evolve, the ability to read and understand the rich tapestry of information within videos will unlock countless applications across various fields.