DRAMA-base
| Property | Value |
|---|---|
| Parameters | 0.1B non-embedding parameters |
| Model Type | Dense Retrieval Model |
| Default Embedding Size | 768 (flexible) |
| Supported Languages | 20 languages including Arabic, Chinese, English, etc. |
| Paper | arXiv:2502.18460 |
What is drama-base?
DRAMA-base is an innovative dense retrieval model that represents a significant advancement in efficient multilingual text retrieval. Derived from pruning a larger language model backbone, it achieves impressive performance while maintaining a compact size of just 0.1B non-embedding parameters. The model implements Matryoshka Representation Learning (MRL), allowing for flexible dimensionality reduction from 768 to smaller dimensions like 512 or 256 without significant performance loss.
Implementation Details
The model utilizes bi-directional attention and implements special query handling by adding "Query: " as a prefix for query text. It can be easily integrated using either the Transformers or Sentence Transformers libraries, making it accessible for various applications.
- Built on a pruned Llama3.2-1B architecture
- Supports flexible embedding dimensions through MRL
- Implements bi-directional attention mechanism
- Specialized query handling with prefix support
Core Capabilities
- Multilingual text retrieval across 20 languages
- Efficient dense retrieval with compact model size
- Flexible dimensionality reduction options
- Strong performance on BEIR, MIRACL, MLDR, and MTEB benchmarks
Frequently Asked Questions
Q: What makes this model unique?
DRAMA-base uniquely combines the power of large language models with efficient dense retrieval, offering flexible dimensionality while maintaining strong performance across multiple languages. Its pruned architecture and MRL implementation make it particularly efficient for practical applications.
Q: What are the recommended use cases?
The model is ideal for multilingual information retrieval tasks, document search, and cross-lingual retrieval applications. It's particularly effective for scenarios requiring efficient retrieval across multiple languages while maintaining compact resource usage.
