Dolphin-LLaMA2-7B
| Property | Value |
|---|---|
| Base Model | LLaMA2-7B |
| License | LLaMA2 |
| Training Data | Orca-style Dataset (3.4M instructions) |
| Primary Use | Text Generation, Instruction Following |
What is dolphin-llama2-7b?
Dolphin-LLaMA2-7B is an uncensored language model based on Meta's LLaMA2 architecture, specifically designed to provide highly compliant and detailed responses. The model was trained on a substantial dataset of 3.4M instructions, combining FLANv2 data augmented with both GPT-4 and GPT-3.5 completions.
Implementation Details
The model underwent a two-phase training process on 8 A400 GPUs over approximately 400 hours. The first phase involved training on 2.6M GPT-3.5 augmented instructions for 3 epochs at 2e-5 learning rate, followed by training on 842K GPT-4 augmented instructions for 2.5 epochs at 1e-5 learning rate.
- Utilizes Vicuna-style prompt format with added SYSTEM field
- Implements Microsoft's Orca training methodology
- Features removed alignment and bias filtering for increased compliance
- Achieves notable benchmark scores including 67.52 on HellaSwag and 48.37 on MMLU
Core Capabilities
- Advanced reasoning and step-by-step explanation generation
- Highly compliant instruction following
- Detailed context provision before answering questions
- Strong performance on complex reasoning tasks
Frequently Asked Questions
Q: What makes this model unique?
The model's distinctive feature is its uncensored nature combined with high compliance, making it suitable for various applications while allowing users to implement their own alignment layer. It's particularly notable for its implementation of Microsoft's Orca methodology with custom modifications to the training process.
Q: What are the recommended use cases?
The model is well-suited for tasks requiring detailed explanations, step-by-step reasoning, and complex instruction following. However, due to its uncensored nature, it's recommended to implement an alignment layer before deployment in production environments.
