| Property | Value |
|---|---|
| Parameter Count | 6.92B total (1B active) |
| Model Type | Mixture-of-Experts Language Model |
| License | Apache 2.0 |
| Paper | arXiv:2409.02060 |
| Tensor Type | BF16 |
What is OLMoE-1B-7B-0924?
OLMoE-1B-7B is a groundbreaking Mixture-of-Experts (MoE) language model that achieves remarkable efficiency by using only 1B active parameters while maintaining access to a total parameter count of 7B. Released by Allen AI in September 2024, it represents a significant advancement in efficient language modeling, delivering performance competitive with much larger models like Llama2-13B while maintaining a smaller active parameter footprint.
Implementation Details
The model implements a sophisticated MoE architecture that intelligently routes computation through specialized expert neural networks. It's trained on a diverse dataset and offers both BF16 and FP32 versions, with the BF16 version being the default due to comparable performance.
- Fully open-source implementation with transparent training logs and code
- Supports multiple fine-tuning approaches including SFT and DPO/KTO
- Includes various checkpoints for different stages of training
- Compatible with the Transformers library (requires installation from source)
Core Capabilities
- State-of-the-art performance on multiple benchmarks (MMLU: 54.1, HellaSwag: 80.0)
- Efficient text generation and processing
- Competitive performance against larger models while using fewer active parameters
- Excellent results on reasoning tasks (ARC-Challenge: 62.1, WinoGrande: 70.2)
Frequently Asked Questions
Q: What makes this model unique?
OLMoE-1B-7B's uniqueness lies in its ability to achieve high performance with only 1B active parameters through its Mixture-of-Experts architecture, making it both efficient and powerful. It's also completely open-source, allowing for transparent research and development.
Q: What are the recommended use cases?
The model is well-suited for general language tasks, including text generation, reasoning, and analysis. Its efficient architecture makes it particularly valuable for deployments where computational resources are constrained but high performance is required.