WordLlama L2 Supercat
| Property | Value |
|---|---|
| License | MIT |
| Language | English |
| Default Model Size | 16MB (256-dim) |
| Author | dleemiller |
What is word-llama-l2-supercat?
WordLlama is an innovative NLP toolkit that recycles components from large language models (LLMs) to create efficient and compact word representations. This particular model, l2_supercat, is built using concatenated codebooks from several models, including Llama2 70B and phi3 medium, resulting in a 32k vocabulary size that's 4x smaller than typical implementations while maintaining comparable performance.
Implementation Details
The model employs Matryoshka representation learning, allowing flexible dimensionality from 64 to 1024 dimensions. It operates through a simple token lookup with average pooling, optimized for CPU usage. The implementation requires only numpy for inference, making it extremely lightweight and portable.
- Supports multiple embedding dimensions (64, 128, 256, 512, 1024)
- 16MB default model size (256-dim)
- Numpy-only inference requirements
- Binary embedding support with hamming distance calculations
Core Capabilities
- Fuzzy-deduplication of text
- Semantic similarity matching
- Document ranking and clustering
- Text embedding generation
- K-means clustering with customizable parameters
- Top-k document retrieval
Frequently Asked Questions
Q: What makes this model unique?
The model's ability to achieve comparable performance to GloVe 300d while being significantly smaller (16MB vs >2GB) and its CPU-optimized design make it stand out. It also offers flexible dimensionality through Matryoshka representations and requires minimal dependencies.
Q: What are the recommended use cases?
WordLlama is ideal for NLP-lite tasks such as training sklearn classifiers, semantic matching, fuzzy deduplication, ranking, and clustering. It's particularly useful for creating LLM output evaluators and preparatory tasks in multi-hop or agentic workflows.
