| Property | Value |
|---|---|
| Research Paper | View Paper |
| Training Data | 80B tokens |
| Training Duration | 14 days on 128 GPUs |
| Perplexity | 13.6 |
| Context Length | 2048 tokens |
What is rugpt3large_based_on_gpt2?
rugpt3large_based_on_gpt2 is a powerful Russian language model developed by the SberDevices team. It represents a significant advancement in Russian natural language processing, trained on a massive dataset of 80B tokens. The model underwent a sophisticated two-phase training process: initial training with 1024 sequence length for 3 epochs, followed by fine-tuning with an extended 2048 sequence length for 1 epoch.
Implementation Details
The model is built on the GPT-2 architecture using the Transformers library, incorporating several technical innovations:
- Extended context length of 2048 tokens for improved long-form text handling
- Comprehensive training on 128 GPUs for the initial phase
- Additional fine-tuning on 16 GPUs for context length extension
- Achieves state-of-the-art perplexity of 13.6 on test sets
Core Capabilities
- Advanced Russian text generation
- Long-form content creation with extended context window
- Robust performance across various text generation tasks
- Optimized for Russian language understanding and generation
Frequently Asked Questions
Q: What makes this model unique?
This model stands out for its extensive training on Russian language data and its sophisticated two-phase training approach, resulting in superior performance for Russian text generation tasks. The extended context length of 2048 tokens allows for more coherent long-form content generation.
Q: What are the recommended use cases?
The model is particularly well-suited for Russian language applications including content generation, text completion, and creative writing tasks. Its extended context length makes it especially valuable for generating longer, coherent texts while maintaining context consistency.