| Property | Value |
|---|---|
| Author | bongsoo |
| Model Type | Cross-Encoder |
| Base Architecture | ALBERT-small-Korean |
| Hugging Face | Model Repository |
What is albert-small-kor-cross-encoder-v1?
This is a specialized Korean language model based on ALBERT architecture, fine-tuned as a cross-encoder for semantic similarity tasks. The model has been systematically trained using a combination of STS (Semantic Textual Similarity) and NLI (Natural Language Inference) datasets, achieving impressive performance scores across multiple benchmarks.
Implementation Details
The model underwent a sophisticated training regime combining STS and NLI training in alternating phases (sts-nli-sts-nli-sts). Training parameters were carefully optimized with specific configurations for both STS (10 epochs, learning rate 1e-4) and NLI (3 epochs, learning rate 3e-5) training phases.
- STS Training: 10 epochs, lr=1e-4, eps=1e-6, warm_step=10%, max_seq_len=128
- NLI Training: 3 epochs, lr=3e-5, eps=1e-8, warm_step=10%, max_seq_len=128
- Achieves state-of-the-art performance on multiple benchmarks: KorSTS (0.8455), KLUE-STS (0.8526), GLUE(STSB) (0.8513)
Core Capabilities
- Semantic similarity scoring for Korean text pairs
- Cross-lingual capabilities with strong performance on English STS tasks
- Efficient inference with small model footprint
- Easy integration with SentenceTransformers framework
Frequently Asked Questions
Q: What makes this model unique?
This model combines the efficiency of ALBERT architecture with specialized training for Korean language understanding, achieving competitive performance while maintaining a smaller model size. The alternating STS-NLI training strategy enables robust semantic understanding.
Q: What are the recommended use cases?
The model is ideal for tasks requiring semantic similarity assessment in Korean text, including content matching, plagiarism detection, and semantic search applications. It can be easily integrated using the SentenceTransformers CrossEncoder class.