orca_mini_13b
| Property | Value |
|---|---|
| Base Model | OpenLLaMA-13B |
| License | CC-BY-NC-SA-4.0 |
| Paper | Orca Paper |
| Average Benchmark Score | 41.36% |
What is orca_mini_13b?
orca_mini_13b is an advanced language model built on OpenLLaMA-13B architecture, fine-tuned using explain-tuned datasets from WizardLM (~70K), Alpaca (~52K), and Dolly-V2 (~15K). The model implements approaches from the Orca Research Paper, focusing on learning thought processes from ChatGPT as a teacher model.
Implementation Details
The model was trained on 8x A100(80G) GPUs for approximately 15 hours using DeepSpeed with fully sharded data parallelism (ZeRO stage 3). The training utilized a batch size of 16, learning rate of 2e-5, and ran for 3 epochs with AdamW optimizer.
- Maximum sequence length: 1024 tokens
- Training micro batch size per GPU: 2
- Gradient accumulation steps: 1
- Implements 15 system instructions from Orca Research
Core Capabilities
- Strong performance on Winogrande (64.17% accuracy)
- Effective on HellaSwag with 63.4% accuracy
- TruthfulQA performance of 43.1%
- MMLU score of 35.43%
- Capable of following complex instructions and generating explanatory responses
Frequently Asked Questions
Q: What makes this model unique?
The model's unique strength lies in its explain-tuned approach, learning from multiple high-quality datasets while implementing Orca research methodologies to capture complex reasoning patterns from ChatGPT.
Q: What are the recommended use cases?
The model is well-suited for text generation tasks, instruction following, and explanation generation. However, it should not be relied upon for factual accuracy and may have limitations in mathematical reasoning (as evidenced by 0% accuracy on GSM8k).
