Zephyr-7B-beta-GPTQ
| Property | Value |
|---|---|
| Parameter Count | 7B |
| Model Type | GPTQ-Quantized LLM |
| Base Model | Mistral-7B-v0.1 |
| License | MIT |
| Research Paper | Zephyr: Direct Distillation of LM Alignment |
What is zephyr-7B-beta-GPTQ?
Zephyr-7B-beta-GPTQ is a quantized version of the Zephyr language model, optimized for efficient deployment while maintaining high performance. It's built on Mistral-7B and fine-tuned using Direct Preference Optimization (DPO) on carefully curated datasets including UltraChat and UltraFeedback.
Implementation Details
This model implements GPTQ quantization with multiple parameter options, including 4-bit and 8-bit versions with different group sizes. The main branch features 4-bit quantization with act-order and 128g group size, optimizing for both efficiency and performance.
- Multiple quantization options available (4-bit to 8-bit)
- Achieves 7.34 score on MT-Bench, outperforming many larger models
- Supports various inference frameworks including text-generation-webui and Hugging Face TGI
Core Capabilities
- High-quality conversational interactions
- Strong performance on general knowledge tasks
- Efficient deployment with reduced memory footprint
- Compatible with major inference frameworks
Frequently Asked Questions
Q: What makes this model unique?
This model stands out for achieving state-of-the-art performance among 7B parameter models while being efficiently quantized for practical deployment. It scores 7.34 on MT-Bench, surpassing many larger models including some 70B parameter versions.
Q: What are the recommended use cases?
The model is best suited for conversational AI applications, general text generation, and assistant-style interactions. It's particularly efficient for deployment in resource-constrained environments while maintaining high-quality outputs.
