AfriBERTa Large
| Property | Value |
|---|---|
| Parameters | 126 million |
| Architecture | 10 layers, 6 attention heads, 768 hidden units |
| License | MIT |
| Languages Supported | 11 African languages |
What is afriberta_large?
AfriBERTa Large is a sophisticated multilingual language model specifically designed for African languages. It represents a significant advancement in Natural Language Processing for low-resource African languages, trained on 11 different languages including Afaan Oromoo, Amharic, Gahuza, Hausa, Igbo, Nigerian Pidgin, Somali, Swahili, Tigrinya, and Yorùbá.
Implementation Details
The model features a robust architecture with 126 million parameters, structured with 10 layers, 6 attention heads, 768 hidden units, and a feed-forward size of 3072. It's built using the Transformers architecture and can be easily integrated using the Hugging Face framework.
- Trained on BBC news and Common Crawl datasets
- Supports both token classification and sequence classification tasks
- Maximum sequence length of 512 tokens
Core Capabilities
- Text Classification across multiple African languages
- Named Entity Recognition (NER)
- Fill-Mask task support
- Cross-lingual transfer learning
Frequently Asked Questions
Q: What makes this model unique?
AfriBERTa Large is specifically optimized for African languages, showing competitive performance even on languages it wasn't pretrained on. It achieves this despite being trained on relatively small datasets (less than 1GB), making it particularly valuable for low-resource languages.
Q: What are the recommended use cases?
The model is best suited for text classification and named entity recognition tasks in African languages. It's particularly valuable for researchers and developers working with low-resource African languages, though users should be aware it was primarily trained on news articles which may limit its domain adaptation.
