OpenAssistant LLaMa 30B SFT 6
| Property | Value |
|---|---|
| Base Architecture | LLaMa 30B |
| License | Other (Meta AI License) |
| Paper | OASST Dataset Paper |
| Training Type | Supervised Fine-Tuning |
What is oasst-sft-6-llama-30b-xor?
This is a sophisticated language model based on Meta's LLaMa 30B architecture, fine-tuned by OpenAssistant using supervised fine-tuning (SFT) techniques. The model employs an innovative XOR weight distribution method to comply with Meta's licensing requirements while maintaining full functionality.
Implementation Details
The model uses a specialized training configuration with FP16 precision and implements several optimization techniques including gradient checkpointing and flash attention. It was trained using a learning rate of 1e-5 with zero-3 optimization, supporting sequences up to 2048 tokens.
- Utilizes custom sampling and flash attention for improved performance
- Implements gradient accumulation over 16 steps
- Trained across multiple high-quality datasets including OASST export, Vicuna, Dolly15k, and code-alpaca
- Employs sophisticated batch processing with 2 samples per device for training
Core Capabilities
- Multilingual support across 20 languages including English, German, French, Spanish, and others
- Specialized handling of mathematical instructions and coding tasks
- Efficient processing of long-form content up to 2048 tokens
- Balanced training across diverse datasets for robust general-purpose use
Frequently Asked Questions
Q: What makes this model unique?
This model stands out for its innovative XOR weight distribution approach, allowing users to legally utilize the model while respecting Meta's licensing terms. It also features extensive multilingual capabilities and specialized training across diverse domains.
Q: What are the recommended use cases?
The model is well-suited for multilingual applications, code generation, mathematical problem-solving, and general-purpose language tasks. Its diverse training data makes it particularly effective for educational and technical content generation.
