MistralTrix-v1
| Property | Value |
|---|---|
| Parameter Count | 8.99B |
| Model Type | Text Generation |
| Architecture | Mistral-based Transformer |
| License | Apache-2.0 |
| Precision | FP16 |
What is MistralTrix-v1?
MistralTrix-v1 is an advanced language model that represents a significant achievement in the field of natural language processing. Built upon the zyh3826/GML-Mistral-merged-v1 architecture, this model has been fine-tuned using Direct Preference Optimization (DPO) with Intel's dataset for neural-chat-7b-v3-1. Notable for achieving top rankings among 7B LLMs, it demonstrates exceptional performance across various benchmarks.
Implementation Details
The model was trained using a sophisticated LoRA configuration with r=16 and lora_alpha=16, implementing key optimizations including gradient checkpointing and a cosine learning rate scheduler. Training was completed on a single Colab GPU with 40GB VRAM in less than a few hours, utilizing FP16 precision and 4-bit quantization.
- Comprehensive target module optimization including k_proj, gate_proj, v_proj, up_proj, q_proj, o_proj, and down_proj
- Batch size of 4 with gradient accumulation steps of 4
- Learning rate of 5e-5 with 100 warmup steps
- Maximum sequence length of 1536 tokens
Core Capabilities
- ARC performance: 72.27%
- HellaSwag score: 88.33%
- MMLU capability: 65.24%
- TruthfulQA accuracy: 70.73%
- Winogrande performance: 80.98%
- GSM8K mathematical reasoning: 62.77%
Frequently Asked Questions
Q: What makes this model unique?
MistralTrix-v1 stands out for its impressive benchmark performances and efficient training approach, achieving state-of-the-art results for its parameter size through DPO fine-tuning on Intel's dataset.
Q: What are the recommended use cases?
The model excels in general text generation tasks, showing particular strength in reasoning, comprehension, and truthfulness evaluation, making it suitable for various NLP applications requiring high-quality text generation and analysis.
