Jamba-v0.1
| Property | Value |
|---|---|
| Parameter Count | 51.6B (12B active) |
| Context Length | 256K tokens |
| License | Apache 2.0 |
| Paper | Research Paper |
| Architecture | Hybrid SSM-Transformer with MoE |
What is Jamba-v0.1?
Jamba-v0.1 is a groundbreaking language model that combines State Space Model (SSM) architecture with traditional Transformers. Developed by AI21, it represents the first production-scale implementation of the Mamba architecture, featuring 51.6B total parameters with 12B active parameters through its mixture-of-experts design.
Implementation Details
The model utilizes a hybrid architecture that leverages both attention mechanisms and Mamba's SSM approach. It supports an impressive 256K context length and can process up to 140K tokens on a single 80GB GPU when using 8-bit quantization.
- Supports both BF16 and F32 precision
- Implements FlashAttention2 for optimized performance
- Includes specialized Mamba kernels for enhanced processing speed
- Features mixture-of-experts architecture for efficient parameter usage
Core Capabilities
- Strong benchmark performance: 87.1% on HellaSwag, 67.4% on MMLU
- GSM8K (CoT) performance of 59.9%
- Efficient processing with optimized Mamba implementation
- Supports fine-tuning for custom applications
Frequently Asked Questions
Q: What makes this model unique?
Jamba-v0.1 is the first production-scale implementation of Mamba architecture, combining it with traditional Transformer elements to achieve better throughput while maintaining competitive performance on standard benchmarks.
Q: What are the recommended use cases?
The model is designed as a base model for fine-tuning and custom solution development. It's particularly suitable for applications requiring long context processing and can be adapted for various downstream tasks through fine-tuning.
