OpenAssistant LLaMA 30B SFT 7
| Property | Value |
|---|---|
| Base Model | LLaMA 30B |
| License | Other |
| Research Paper | OASST Dataset Paper |
| Training Approach | Supervised Fine-Tuning |
What is oasst-sft-7-llama-30b-xor?
The oasst-sft-7-llama-30b-xor is a sophisticated language model based on Meta's LLaMA 30B architecture, fine-tuned by OpenAssistant using supervised learning techniques. Due to LLaMA's licensing restrictions, the model is distributed using an innovative XOR weights approach, requiring users to combine it with the original LLaMA weights.
Implementation Details
The model implementation requires a specific process using XOR decoding, with strict version requirements for dependencies including Python 3.10, PyTorch 1.13.1, and specific versions of transformers and other libraries. The model was trained using a combination of datasets including OASST export, Vicuna, Dolly15k, and others, with specific configuration parameters such as FP16 dtype and flash attention enabled.
- Utilizes gradient checkpointing and custom sampling
- Implements flash attention for improved performance
- Trained with a learning rate of 1e-5 and zero weight decay
- Maximum sequence length of 2048 tokens
Core Capabilities
- Multilingual support across 20 languages
- Handles various task types including instruction following
- Optimized for both general conversation and specialized tasks
- Supports code generation and mathematical reasoning
Frequently Asked Questions
Q: What makes this model unique?
This model uniquely combines the powerful LLaMA 30B architecture with OpenAssistant's supervised fine-tuning approach, distributed through an innovative XOR weights system to comply with licensing requirements.
Q: What are the recommended use cases?
The model is suitable for multilingual applications, instruction following, code generation, and mathematical problem-solving. It's particularly effective for tasks requiring understanding across multiple languages and domains.
