Dolphin 2.6 Phi-2 GGUF
| Property | Value |
|---|---|
| Parameter Count | 2.78B |
| License | Microsoft Research License (Non-commercial) |
| Base Model | Microsoft Phi-2 |
| Training Datasets | 7 specialized datasets including Dolphin, Airoboros, OpenHermes, and Magicoder |
What is dolphin-2_6-phi-2-GGUF?
Dolphin 2.6 Phi-2 GGUF is a quantized version of the Dolphin 2.6 model, based on Microsoft's Phi-2 architecture. This model represents a significant advancement in compact AI assistants, offering various quantization options from 2-bit to 8-bit to balance performance and resource usage. The model was trained over 3 epochs on 4x A100s using qLoRA and Axolotl frameworks.
Implementation Details
The model uses the ChatML prompt format and comes in multiple GGUF variants optimized for different use cases. The quantization options range from Q2_K (1.17GB) to Q8_0 (2.96GB), allowing users to choose based on their hardware constraints and quality requirements.
- Multiple quantization options (Q2_K through Q8_0)
- Supports GPU acceleration with layer offloading
- Compatible with llama.cpp and various UI frameworks
- Uses ChatML prompt format for consistent interactions
Core Capabilities
- General conversational AI tasks
- Enhanced structured output generation
- Compliant response generation (note: uncensored model)
- Context window of 2048 tokens
- Multi-platform compatibility (CPU & GPU)
Frequently Asked Questions
Q: What makes this model unique?
This model combines Microsoft's Phi-2 architecture with specialized training on multiple high-quality datasets, offering a balance between model size and performance. Its various quantization options make it accessible across different hardware configurations.
Q: What are the recommended use cases?
The model is well-suited for conversational AI applications, text generation, and structured output tasks. For optimal performance-to-size ratio, the Q4_K_M quantization is recommended for most users.
