OpenOrca-Preview1-13B
| Property | Value |
|---|---|
| Base Model | LLaMA-13B |
| License | MIT |
| Training Cost | < $200 |
| Training Infrastructure | 8x A100-80G GPUs |
| Training Duration | 15 hours |
What is OpenOrca-Preview1-13B?
OpenOrca-Preview1-13B is an innovative language model that fine-tunes LLaMA-13B using a carefully curated subset of the OpenOrca dataset. This preview release demonstrates remarkable efficiency by achieving 60% of the improvements shown in Microsoft's Orca paper while using only 6% of the available training data. The model was trained on 200,000 filtered GPT-4 entries, with specific attention to removing potentially harmful patterns like "As an AI language model" statements.
Implementation Details
The model leverages the Axolotl training framework and employs the Alpaca prompt format. Training was conducted over 4 epochs, with the best-performing snapshot selected at the 3-epoch mark. The model demonstrates impressive performance metrics, scoring 0.3753 on BigBench-Hard and 0.3638 on AGIEval benchmarks.
- Built with PyTorch and Transformers library
- Implements text-generation-inference capabilities
- Trained on filtered, high-quality GPT-4 data
- Optimized for reasoning and complex task handling
Core Capabilities
- Advanced reasoning abilities demonstrated through benchmark performance
- Efficient text generation and processing
- Handles complex instructions and explanations
- Optimized for English language tasks
Frequently Asked Questions
Q: What makes this model unique?
This model achieves remarkable performance metrics while using only a fraction of the training data, demonstrating exceptional efficiency in learning from high-quality, filtered datasets. It represents a significant step forward in cost-effective model training while maintaining high performance standards.
Q: What are the recommended use cases?
The model is particularly well-suited for complex reasoning tasks, text generation, and applications requiring sophisticated language understanding. It performs especially well on benchmark tasks from BigBench-Hard and AGIEval, making it suitable for academic and research applications.
