rubert-tiny2
| Property | Value |
|---|---|
| Parameter Count | 29.4M |
| License | MIT |
| Language | Russian |
| Vocabulary Size | 83,828 tokens |
| Max Sequence Length | 2048 tokens |
What is rubert-tiny2?
rubert-tiny2 is an enhanced version of the rubert-tiny model, specifically designed for Russian language processing. It's a lightweight BERT-based encoder that produces high-quality sentence embeddings while maintaining a relatively small parameter footprint of 29.4M parameters. The model represents a significant improvement over its predecessor, featuring an expanded vocabulary and increased sequence length support.
Implementation Details
The model can be easily implemented using either the transformers library with PyTorch or the sentence_transformers framework. It supports various text processing tasks and has been optimized to approximate LaBSE (Language-agnostic BERT Sentence Embeddings) more closely than its previous version.
- Expanded vocabulary of 83,828 tokens (up from 29,564)
- Extended sequence length support up to 2048 tokens
- Improved LaBSE approximation capabilities
- Tuned segment embeddings through NLI task optimization
- Russian language specialization
Core Capabilities
- Sentence similarity computation
- Feature extraction for text analysis
- Masked language modeling
- Text embeddings generation
- KNN classification for short texts
- Fine-tuning support for downstream tasks
Frequently Asked Questions
Q: What makes this model unique?
The model combines compact size with high performance for Russian language tasks, featuring significantly expanded vocabulary and sequence length capabilities compared to its predecessor. It's specifically optimized for generating high-quality sentence embeddings while maintaining efficiency.
Q: What are the recommended use cases?
The model is ideal for tasks requiring sentence embeddings in Russian text processing, including text similarity comparison, classification tasks, and as a foundation for fine-tuning on specific downstream tasks. It's particularly suitable for applications where computational resources are limited but high-quality Russian language understanding is required.
