MiniCPM3-RAG-LoRA
| Property | Value |
|---|---|
| Parameter Count | 4B |
| Max Input Tokens | 32,768 |
| Languages | Chinese, English |
| License | Apache-2.0 (code), Custom MiniCPM License (weights) |
What is MiniCPM3-RAG-LoRA?
MiniCPM3-RAG-LoRA is a specialized language model developed through collaboration between ModelBest Inc., NEUIR, and THUNLP, specifically designed for Retrieval-Augmented Generation (RAG) scenarios. Built upon the MiniCPM3 foundation model, it employs Low-Rank Adaptation (LoRA) and Direct Preference Optimization (DPO) techniques, trained on over 20,000 open-domain QA and logical reasoning examples.
Implementation Details
The model leverages advanced fine-tuning techniques to achieve superior performance in RAG applications. It requires transformers>=4.36.0 and supports both Chinese and English languages.
- Implements LoRA architecture for efficient fine-tuning
- Supports extensive context window of 32,768 tokens
- Utilizes DPO for enhanced preference alignment
- Achieves 13% performance improvement over baseline
Core Capabilities
- Open-domain question answering (NQ, TQA, MARCO)
- Multi-hop reasoning (HotpotQA)
- Dialogue generation (WoW)
- Fact verification (FEVER)
- Information filling (T-REx)
- Outperforms larger models like Llama3-8B and Baichuan2-13B
Frequently Asked Questions
Q: What makes this model unique?
The model's specialized RAG optimization and impressive performance metrics despite its relatively small size (4B parameters) make it stand out. It achieves state-of-the-art results across multiple benchmarks while maintaining efficiency.
Q: What are the recommended use cases?
The model excels in scenarios requiring document retrieval and generation, including question answering, fact checking, and information synthesis. It's particularly suitable for applications needing strong reasoning capabilities with external knowledge integration.
