SorcererLM-8x22b-bf16
| Property | Value |
|---|---|
| Parameter Count | 141B |
| Precision | BF16 |
| License | Apache 2.0 |
| Base Model | WizardLM-2-8x22B |
What is SorcererLM-8x22b-bf16?
SorcererLM-8x22b-bf16 is an advanced language model that builds upon the WizardLM-2-8x22B architecture using low-rank LoRA training. It's specifically designed to enhance roleplay capabilities while maintaining the base model's intelligence. The model employs a low-rank (r=16, alpha=32) 16bit-LoRA approach, trained on cleaned and deduplicated c2-logs for 2 epochs.
Implementation Details
The model utilizes a carefully chosen LoRA implementation instead of FFT, focusing on improving vocabulary and writing style for roleplay scenarios while preserving WizardLM's core capabilities. Training was conducted using qlora-pipe, with specific attention to maintaining optimal performance.
- Low-rank LoRA implementation (r=16, alpha=32)
- BF16 precision for efficient computing
- Compatible with Vicuna 1.1 prompting format
- Optimized for shorter prompts
Core Capabilities
- Enhanced roleplay performance compared to base model
- Improved vocabulary and writing style
- Efficient processing with BF16 precision
- Compatible with various quantized versions (iMat GGUFs and longcal exl2s)
Frequently Asked Questions
Q: What makes this model unique?
The model's unique approach lies in its use of LoRA training to enhance roleplay capabilities while maintaining the base model's intelligence. It specifically addresses vocabulary limitations in roleplay scenarios while preserving WizardLM's core strengths.
Q: What are the recommended use cases?
The model is optimized for roleplay scenarios and performs best with shorter prompts. It's recommended to use templates from Quant-Cartel/Recommended-Settings under the SorcererLM folder, or Vicuna 1.1 with a sensible context template. Optimal results are achieved with Temperature 1, MinP 0.05, and DRY sampling parameters.
