DRAMA-large
| Property | Value |
|---|---|
| Model Size | 0.3B parameters |
| Embedding Size | 1024 (flexible) |
| Author | |
| Paper | arXiv:2502.18460 |
| Supported Languages | 20 languages |
What is drama-large?
DRAMA-large is a sophisticated dense retrieval model derived from pruning a large language model backbone. With 0.3B non-embedding parameters, it represents a compact yet powerful solution for multilingual text retrieval. The model leverages Matryoshka Representation Learning (MRL) to provide flexible embedding dimensionality, allowing users to efficiently encode text in dimensions ranging from 1024 down to smaller sizes like 256.
Implementation Details
The model is built upon a pruned version of Llama3.2-1B and features several technical innovations:
- Bi-directional attention mechanism instead of uni-directional attention
- Query prefix enhancement with "Query: " for improved retrieval performance
- Flexible dimension truncation through Matryoshka Representation Learning
- Support for both Transformers and Sentence Transformers implementations
Core Capabilities
- Multilingual text retrieval across 20 languages including Arabic, Chinese, English, and more
- Flexible embedding dimensionality (1024, 512, 256)
- Strong performance on benchmarks like BEIR, MIRACL, MLDR, and MTEB
- Efficient retrieval with compact model size
- Seamless integration with popular deep learning frameworks
Frequently Asked Questions
Q: What makes this model unique?
DRAMA-large stands out for its efficient architecture that combines the power of large language models with the practicality of dense retrieval. Its unique Matryoshka Representation Learning allows for flexible dimensionality while maintaining performance, making it highly adaptable to different computational constraints.
Q: What are the recommended use cases?
The model is ideal for multilingual information retrieval tasks, document search, and cross-lingual retrieval applications. It's particularly effective for organizations needing efficient, high-quality retrieval across multiple languages while maintaining computational efficiency.
