Komodo-7B-Base
| Property | Value |
|---|---|
| Model Type | Decoder (Llama-2 Architecture) |
| Parameters | 7 Billion |
| Vocabulary Size | 35,008 tokens |
| Sequence Length | 4096 |
| License | Llama2 |
| Paper | arXiv:2403.09362 |
What is komodo-7b-base?
Komodo-7B-Base is an innovative large language model developed by Yellow.ai through incremental pretraining and vocabulary expansion on top of Llama-2-7B-Base. The model is specifically designed to handle multiple languages, including Indonesian, English, and 11 regional Indonesian languages, making it a powerful tool for multilingual applications in the Indonesian linguistic landscape.
Implementation Details
The model features a sophisticated architecture with 32 layers and a dimension size of 4096. Its unique tokenizer implementation includes approximately 2,000 frequently used Indonesian words and 1,000 regional language words that were absent in the original Llama-2 model. The training was conducted on AWS EC2 p4d.24xlarge instances using 8 Nvidia A100 40GB GPUs over 300 hours.
- Custom decoding function requiring trust_remote_code=True
- Enhanced vocabulary focusing on Indonesian linguistic diversity
- Optimized transformer architecture based on Llama-2
- Comprehensive support for 13 languages including regional dialects
Core Capabilities
- Multilingual text generation and understanding
- Enhanced processing of Indonesian and regional languages
- Base model capabilities for further fine-tuning
- 4096 token context window
- Optimized for Indonesian language tasks
Frequently Asked Questions
Q: What makes this model unique?
The model's distinctive feature is its specialized vocabulary expansion for Indonesian and regional languages, combined with its foundation on the Llama-2 architecture. It's specifically designed to handle multiple Indonesian dialects while maintaining English language capabilities.
Q: What are the recommended use cases?
As a base model, it's ideal for further fine-tuning on specific tasks. It's particularly suited for applications requiring understanding of Indonesian and its regional languages, though it requires additional instruction tuning for specific downstream tasks.
