Rocket-3B
| Property | Value |
|---|---|
| Parameter Count | 2.8B |
| Base Model | StableLM-3B-4e1t |
| Training Method | Direct Preference Optimization (DPO) |
| License | CC-BY-SA-4.0 |
| Primary Language | English |
| MT-Bench Score | 6.56 |
What is Rocket-3B?
Rocket-3B is a compact yet powerful language model that demonstrates exceptional performance despite its relatively small size. Named after the character from 'Guardians of the Galaxy', this 3B parameter model leverages Direct Preference Optimization (DPO) training to achieve performance metrics that rival much larger models. Built on the StableLM architecture, it particularly excels in chat-based applications using the ChatML format.
Implementation Details
The model implements a sophisticated architecture utilizing FP16 precision and incorporates advanced training techniques. It uses the ChatML format for input processing and can be easily integrated using the Hugging Face Transformers library. The model was trained on a curated mixture of public datasets, including Falcon RefinedWeb, RedPajama-Data, and StarCoder.
- Achieves 6.99 score in first-turn and 6.13 in second-turn MT-Bench evaluations
- 79.75% win rate in AlpacaEval with 1,242 average token length
- Implements efficient FP16 tensor operations
- Supports streaming text generation with customizable parameters
Core Capabilities
- Advanced chat functionality with ChatML format support
- Competitive performance in reasoning and language understanding tasks
- Efficient text generation with customizable sampling parameters
- Strong performance in benchmarks like MMLU (47.10%) and HellaSwag (76.69%)
- Effective handling of multi-turn conversations
Frequently Asked Questions
Q: What makes this model unique?
Rocket-3B stands out for achieving impressive performance metrics despite its compact 3B parameter size, often matching or exceeding the capabilities of models 2-20 times larger. Its DPO training approach and efficient architecture enable it to achieve an MT-Bench score of 6.56, surpassing many larger models like Falcon-40B-Instruct.
Q: What are the recommended use cases?
The model is particularly well-suited for chat applications, general text generation, and tasks requiring reasoning capabilities. It performs well in both single-turn and multi-turn conversations, making it ideal for chatbots, content generation, and interactive applications requiring natural language understanding.
