LLaVA-NeXT-Video-7B-hf
| Property | Value |
|---|---|
| Parameter Count | 7.06B |
| Model Type | Video-Text-to-Text |
| License | LLAMA 2 Community License |
| Paper | Research Paper |
| Base Model | lmsys/vicuna-7b-v1.5 |
What is LLaVA-NeXT-Video-7B-hf?
LLaVA-NeXT-Video-7B-hf is an advanced multimodal AI model that combines video and image understanding capabilities. Built on top of LLaVa-NeXT, it represents the current state-of-the-art among open-source models on the VideoMME benchmark. The model processes videos by sampling 32 frames per clip uniformly, enabling comprehensive video analysis and understanding.
Implementation Details
The model has been trained on an extensive dataset comprising both image and video data. The training data includes 558K filtered image-text pairs, 158K GPT-generated instructions, 500K academic VQA data, 50K GPT-4V data, 40K ShareGPT data, and 100K VideoChatGPT-Instruct samples.
- Supports multi-visual and multi-prompt generation
- Handles both image and video inputs simultaneously
- Implements Flash-Attention 2 for improved performance
- Available in 4-bit quantization through bitsandbytes
Core Capabilities
- Video understanding and analysis
- Image-text processing
- Multi-modal instruction following
- Batch processing of mixed media types
- Efficient inference with optimization options
Frequently Asked Questions
Q: What makes this model unique?
The model's ability to process both videos and images in a single architecture, along with its state-of-the-art performance on VideoMME benchmark, sets it apart from other multimodal models. Its flexible architecture allows for multiple input types and efficient processing through various optimization techniques.
Q: What are the recommended use cases?
The model is ideal for video analysis, image understanding, multimodal chatbots, content description, and academic research. It excels in scenarios requiring both video and image processing capabilities, making it suitable for applications in content analysis, education, and research.
