Mistral-7B-OpenOrca-GGUF
| Property | Value |
|---|---|
| Parameter Count | 7.24B |
| License | Apache 2.0 |
| Base Model | Mistral-7B |
| Training Dataset | OpenOrca |
| Paper | Orca Paper |
What is Mistral-7B-OpenOrca-GGUF?
Mistral-7B-OpenOrca-GGUF is a quantized version of the OpenOrca-tuned Mistral language model, offering exceptional performance while maintaining efficiency. This model ranks #2 for all models smaller than 30B at release time, outperforming most 13B models. It uses the GGUF format, which is the successor to GGML, providing various quantization options for different performance and size trade-offs.
Implementation Details
The model supports multiple quantization levels from Q2 to Q8, with file sizes ranging from 3.08GB to 7.70GB. It utilizes ChatML format for input prompts and can be deployed using various frameworks including llama.cpp, text-generation-webui, and Python libraries like ctransformers.
- Multiple quantization options (Q2_K through Q8_0) for different use cases
- Compatible with GPU acceleration through layer offloading
- Supports context window of up to 8K tokens
- Implements ChatML prompt format with system and user messages
Core Capabilities
- Strong performance on MMLU (61.73), ARC (63.57), and HellaSwag (83.79)
- Efficient inference on consumer-grade hardware
- Flexible deployment options across different platforms
- Advanced reasoning capabilities inherited from OpenOrca training
Frequently Asked Questions
Q: What makes this model unique?
This model combines the efficiency of Mistral's architecture with OpenOrca's training methodology, offering near-SOTA performance in a compact, highly-efficient package. The GGUF format allows for various quantization options, making it adaptable to different hardware constraints.
Q: What are the recommended use cases?
The model is well-suited for general language tasks, reasoning, and dialogue applications. For optimal performance-to-size ratio, the Q4_K_M quantization is recommended, while Q5_K_M offers better quality with a larger size footprint.
