Liberated-Qwen1.5-72B
| Property | Value |
|---|---|
| Base Model | Qwen1.5-72B |
| Context Length | 32k tokens (8k for training) |
| License | tongyi-qianwen |
| Training Duration | 3 days on 8x H100s |
| Training Method | qLoRA, deepspeed zero-2 |
What is Liberated-Qwen1.5-72B?
Liberated-Qwen1.5-72B is an advanced language model developed by AbacusAI and Eric Hartford, built upon the Qwen1.5-72B architecture. The model has been specifically finetuned to excel at maintaining system prompt compliance over extended multi-turn conversations, addressing a common limitation in open-source models. It uses the ChatML format and incorporates several key datasets, including the novel SystemChat dataset.
Implementation Details
The model was trained using Axolotl framework with qLoRA optimization, requiring 3 days to complete 3 epochs on 8 H100 GPUs. It maintains a learning rate of 2e-4 and implements deepspeed zero-2 for efficient training. The model demonstrates strong performance, achieving an MMLU score of 77.13 and competitive MT Bench scores compared to the base model.
- Trained on multiple datasets including OpenHermes-2.5, Code-Feedback, and SystemChat
- Implements 32k context window with 8k sequence length during training
- Uses ChatML prompt format for structured interactions
- No built-in guardrails or censorship
Core Capabilities
- Enhanced system prompt compliance over long conversations
- Strong performance in multi-turn dialogues
- Capable of handling mechanical and unusual system prompts
- Maintains high performance on standard benchmarks
- Flexible deployment options with text-generation-inference support
Frequently Asked Questions
Q: What makes this model unique?
The model's primary distinction lies in its enhanced ability to maintain system prompt compliance over long conversations, achieved through specialized training on the SystemChat dataset. It combines the powerful base architecture of Qwen1.5-72B with improved instruction following capabilities.
Q: What are the recommended use cases?
The model is suitable for applications requiring long-form conversations, complex system instructions, and scenarios where consistent adherence to system prompts is crucial. However, users should implement their own alignment layer before deploying it as a service.
