StableBeluga-13B
| Property | Value |
|---|---|
| Developer | Stability AI |
| Base Model | Llama2 13B |
| Training Type | Supervised Fine-tuning |
| License | STABLE BELUGA NON-COMMERCIAL COMMUNITY LICENSE |
| Primary Language | English |
What is StableBeluga-13B?
StableBeluga-13B is an advanced language model developed by Stability AI, built upon the Llama2 13B architecture and fine-tuned using a specialized Orca-style dataset. This model represents a significant step forward in instruction-following AI systems, designed to provide safe and helpful responses while maintaining high performance across various tasks.
Implementation Details
The model utilizes a sophisticated training procedure involving mixed-precision (BF16) training and AdamW optimization. The training process was conducted in two phases: an initial packed training with a batch size of 256 and a subsequent unpacked phase with a batch size of 512. Both phases employed a cosine learning rate decay from 3e-5 to 3e-6, with 100 warm-up steps and weight decay of 1e-6.
- Implemented using HuggingFace Transformers library
- Supports efficient inference with automatic device mapping
- Utilizes a specific prompt format with System, User, and Assistant sections
- Optimized for low CPU memory usage during inference
Core Capabilities
- Advanced instruction following and task completion
- Safe and controlled response generation
- Efficient processing of complex prompts
- Support for diverse conversation formats
- Memory-efficient operation with torch.float16 precision
Frequently Asked Questions
Q: What makes this model unique?
StableBeluga-13B stands out for its careful optimization on an Orca-style dataset, combining the powerful Llama2 architecture with specialized training for improved instruction following. The model's focus on safety and controlled responses makes it particularly suitable for research and development applications.
Q: What are the recommended use cases?
The model is best suited for research and non-commercial applications requiring strong instruction-following capabilities. It excels in scenarios requiring careful, controlled responses while maintaining high performance in language understanding and generation tasks.
