Teuken-7B-instruct-commercial-v0.4
| Property | Value |
|---|---|
| Parameter Count | 7.45B |
| License | Apache 2.0 |
| Research Paper | arXiv:2410.03730 |
| Languages Supported | 24 European Languages |
| Training Data | 4T tokens |
What is Teuken-7B-instruct-commercial-v0.4?
Teuken-7B-instruct-commercial-v0.4 is a multilingual large language model specifically designed for European languages. Developed by the OpenGPT-X project, it has been pre-trained on 4 trillion tokens and fine-tuned for instruction following across 24 official European languages. The model represents a significant step toward reducing English-centric bias in AI language models while better reflecting European values and linguistic diversity.
Implementation Details
The model utilizes a transformer-based decoder-only architecture with 32 layers, 4096 hidden size, and 32 attention heads. It implements group query attention with 2 query groups and uses the SwiGLU activation function. The model was trained using mixed precision BF16 and supports a context length of 4096 tokens.
- Architecture: Transformer decoder-only with 32 layers
- Position Embeddings: Rotary
- Training Objective: Causal Language Modeling (CLM)
- Optimizer: AdamW with learning rates between 3e-4 and 3e-5
Core Capabilities
- Multilingual understanding and generation across 24 European languages
- Instruction following with specialized prompt templates
- Commercial usage support under Apache 2.0 license
- Competitive performance on multilingual benchmarks including EU21-ARC, EU21-HeSw, EU21-TQA, and EU21-MMLU
Frequently Asked Questions
Q: What makes this model unique?
The model's primary distinction lies in its comprehensive coverage of European languages and its specific optimization for European linguistic and cultural context, making it more suitable for European applications compared to English-centric models.
Q: What are the recommended use cases?
The model is particularly well-suited for multilingual tasks across European languages, including text generation, understanding, and instruction following. However, it is not recommended for math and coding tasks.
