LLM2Vec-Sheared-LLaMA-mntp-unsup-simcse
| Property | Value |
|---|---|
| License | MIT |
| Paper | arXiv:2404.05961 |
| Primary Task | Text Embedding Generation |
| Language | English |
What is LLM2Vec-Sheared-LLaMA-mntp-unsup-simcse?
LLM2Vec-Sheared-LLaMA-mntp-unsup-simcse is an innovative text encoder that transforms the LLaMA language model into a powerful embedding generator. It implements a three-step process: enabling bidirectional attention, masked next token prediction, and unsupervised contrastive learning to create high-quality text representations.
Implementation Details
The model combines multiple advanced techniques: it first modifies the traditional decoder-only architecture to support bidirectional attention, then applies masked next token prediction (MNTP) using LoRA weights, and finally incorporates unsupervised SimCSE training. The implementation allows for efficient text encoding with customizable pooling strategies and supports maximum sequence lengths of 512 tokens.
- Bidirectional attention modification for enhanced context understanding
- MNTP LoRA weights merged into base model
- Unsupervised SimCSE training for improved embeddings
- Efficient mean pooling strategy
- Support for instruction-based encoding
Core Capabilities
- High-quality text embedding generation
- Semantic similarity computation
- Information retrieval and document matching
- Classification and clustering tasks
- Cross-document similarity analysis
Frequently Asked Questions
Q: What makes this model unique?
The model's unique approach lies in its ability to convert decoder-only LLMs into effective text encoders through a simple yet powerful three-step process, achieving state-of-the-art performance in various text similarity and retrieval tasks.
Q: What are the recommended use cases?
The model excels in semantic search, document similarity comparison, text classification, and information retrieval tasks. It's particularly effective when instruction-based query encoding is needed for specialized search scenarios.
