BLOOM-1b1 Language Model
| Property | Value |
|---|---|
| Parameter Count | 1.07 Billion |
| Model Type | Transformer-based LLM |
| License | BLOOM RAIL 1.0 |
| Languages | 46 (including 45 natural languages) |
| Framework | PyTorch, JAX, ONNX |
What is BLOOM-1b1?
BLOOM-1b1 is part of the BigScience Large Open-science Open-access Multilingual (BLOOM) family of language models. It's a 1.07B parameter decoder-only transformer model designed for multilingual text generation and understanding. The model supports an impressive array of 46 languages, including 45 natural languages and programming languages, making it one of the most linguistically diverse models of its size.
Implementation Details
The model utilizes a modified Megatron-LM GPT2 architecture with several key optimizations. It features 24 layers, 16 attention heads, and hidden layers of 1536 dimensions. The model implements ALiBI positional encodings and uses GeLU activation functions, with a sequence length of 2048 tokens.
- Architecture: Decoder-only with layer normalization and StableEmbedding
- Training Infrastructure: Trained on Jean Zay Supercomputer
- Tokenization: Byte-level BPE with 250,680 vocabulary size
- Training Data: 1.5TB of pre-processed text across multiple languages
Core Capabilities
- Multilingual text generation in 46 languages
- Zero-shot learning capabilities
- Code generation support
- Text completion and generation tasks
- Base model for fine-tuning on specific tasks
Frequently Asked Questions
Q: What makes this model unique?
BLOOM-1b1's key differentiator is its extensive language support and open-science approach. It's one of the few models of its size that supports such a wide range of languages, including many low-resource African languages.
Q: What are the recommended use cases?
The model is best suited for research purposes, text generation tasks, and as a base model for fine-tuning. However, it should not be used for high-stakes decisions or critical applications without proper evaluation and safeguards.
