| Property | Value |
|---|---|
| Base Model | LLaMA2 70B |
| License | LLAMA 2 Community License |
| Primary Language | English (with multilingual capabilities) |
| Training Type | Two-stage fine-tuning |
What is llama2-70b-oasst-sft-v10?
llama2-70b-oasst-sft-v10 is an advanced fine-tuned version of Meta's LLaMA2 70B model, developed by OpenAssistant. The model underwent a sophisticated two-stage training process, combining synthetic instructions and coding tasks with high-quality human demonstrations from open-assistant.io.
Implementation Details
The model employs a causal decoder-only transformer architecture and uses the chatml prompt format for greater compatibility with existing applications. It was trained using the epfLLM/Megatron-LLM framework, with specific optimizations for performance and reliability.
- Supports context length of 4096 tokens
- Uses flash attention and rotary position embeddings
- Implements two-stage training with pretrain-mix (12k steps) and polish stage (519 steps)
- Compatible with text-generation-inference (TGI) for deployment
Core Capabilities
- Multi-lingual support across 10+ European languages
- Advanced instruction following and coding tasks
- Optimized for helpful, respectful, and honest responses
- Enhanced performance through two-stage fine-tuning process
Frequently Asked Questions
Q: What makes this model unique?
The model's distinctive feature is its two-stage training approach, combining synthetic data with human demonstrations, and its adoption of the chatml format for broader compatibility with existing systems.
Q: What are the recommended use cases?
The model excels in instruction following, coding tasks, and multi-lingual interactions. It's particularly suited for applications requiring helpful, ethical, and accurate responses while maintaining safety guidelines.